羊驼系列大模型和ChatGPT差多少？详细测评后，我沉默了

equation = 'x^2 + 3.0x = 0'roots = [0, -3]answer_gpt = find_roots (llm=chatgpt, equation=equation)

answer_vicuna = find_roots (llm=vicuna, equation=equation)

answer_mpt = find_roots (llm=mpt, equation=equation)

╔═══════════╦══════════╦║   Model   ║ Accuracy ║     ╠═══════════╬══════════╬║ ChatGPT   ║   80%    ║║ Vicuna    ║    0%    ║ ║ MPT       ║    0%    ║╚═══════════╩══════════╩

╔═══════════╦══════════╦║   Model   ║ Accuracy ║     ╠═══════════╬══════════╬║ ChatGPT   ║   100%   ║║ Vicuna    ║    85%   ║ ║ MPT       ║    30%   ║╚═══════════╩══════════╩

qa_attempt1 = guidance ('''{{#system~}}{{llm.default_system_prompt}}{{~/system}}

{{#user~}}You will read a meeting transcript, then extract the relevant segments to answer the following question:Question: {{query}}Here is a meeting transcript:----{{transcript}}----Please answer the following question:Question: {{query}}Extract from the transcript the most relevant segments for the answer, and then answer the question.{{/user}}

{{#assistant~}}{{gen 'answer'}}{{~/assistant~}}''')

qa_attempt3 = guidance ('''{{#system~}}{{llm.default_system_prompt}}{{~/system}}

{{#user~}}You will read a meeting transcript, then extract the relevant segments to answer the following question:Question: {{query}}Here is a meeting transcript:----{{transcript}}----Based on the above, please answer the following question:Question: {{query}}Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.Note that conversation segments can be of any length, e.g. including multiple conversation turns.Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.

As an example of output format, here is a fictitious answer to a question about another meeting transcript.CONVERSATION SEGMENTS:Segment 1: Peter and John discuss the weather.Peter: John, how is the weather today?John: It's raining.Segment 2: Peter insults JohnPeter: John, you are a bad person.Segment 3: BlankANSWER: Peter and John discussed the weather and Peter insulted John.{{/user}}

{{#assistant~}}{{gen 'answer'}}{{~/assistant~}}''')

qa_attempt5 = guidance ('''{{#system~}}{{llm.default_system_prompt}}{{~/system}}

{{#user~}}You will read a meeting transcript, then extract the relevant segments to answer the following question:Question: What were the main things that happened in the meeting?Here is a meeting transcript:----Peter: HeyJohn: HeyPeter: John, how is the weather today?John: It's raining.Peter: That's too bad. I was hoping to go for a walk later.John: Yeah, it's a shame.Peter: John, you are a bad person.----Based on the above, please answer the following question:Question: {{query}}Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.Note that conversation segments can be of any length, e.g. including multiple conversation turns.Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.{{/user}}{{#assistant~}}CONVERSATION SEGMENTS:Segment 1: Peter and John discuss the weather.Peter: John, how is the weather today?John: It's raining.Segment 2: Peter insults JohnPeter: John, you are a bad person.Segment 3: BlankANSWER: Peter and John discussed the weather and Peter insulted John.{{~/assistant~}}{{#user~}}You will read a meeting transcript, then extract the relevant segments to answer the following question:Question: {{query}}Here is a meeting transcript:----{{transcript}}----Based on the above, please answer the following question:Question: {{query}}Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.Note that conversation segments can be of any length, e.g. including multiple conversation turns.Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.{{~/user}}

{{#assistant~}}{{gen 'answer'}}{{~/assistant~}}''')

qa_guided = guidance ('''{{#system~}}{{llm.default_system_prompt}}{{~/system}}

{{#user~}}You will read a meeting transcript, then extract the relevant segments to answer the following question:Question: {{query}}----{{transcript}}----Based on the above, please answer the following question:Question: {{query}}Please extract the three segment from the transcript that are the most relevant for the answer, and then answer the question.Note that conversation segments can be of any length, e.g. including multiple conversation turns. If you need less than three segments, you can leave the rest blank.

As an example of output format, here is a fictitious answer to a question about another meeting transcript:CONVERSATION SEGMENTS:Segment 1: Peter and John discuss the weather.Peter: John, how is the weather today?John: It's raining.Segment 2: Peter insults JohnPeter: John, you are a bad person.Segment 3: BlankANSWER: Peter and John discussed the weather and Peter insulted John.{{/user}}

{{#assistant~}}CONVERSATION SEGMENTS:Segment 1: {{gen'segment1'}}Segment 2: {{gen'segment2'}}Segment 3: {{gen'segment3'}}ANSWER: {{gen 'answer'}}{{~/assistant~}}''')

terminal = guidance ('''{{#system~}}{{llm.default_system_prompt}}{{~/system}}

{{#user~}}Please complete the following task:Task: list the files in the current directoryYou can give me one bash command to run at a time, using the syntax:COMMAND: commandI will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.{{/user}}

{{#assistant~}}COMMAND: ls{{~/assistant~}}

{{#user~}}Output: guidance project{{/user}}

{{#assistant~}}The files or folders in the current directory are:- guidance- projectDONE{{~/assistant~}}

{{#user~}}Please complete the following task:Task: {{task}}You can give me one bash command to run at a time, using the syntax:COMMAND: commandI will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.{{/user}}

{{#geneach 'commands' stop=False}}{{#assistant~}}{{gen 'this.command'}}{{~/assistant~}}

{{~#user~}}Output: {{shell this.command)}}{{~/user~}}{{/geneach}}''')

answer_vicuna = find_roots (llm=vicuna, equation=equation)0