AlgorithmDesign(AD)iscrucialforeffectiveproblem-solvingacrossvariousdomains.TheadventofLargeLanguageModels(LLMs)hasnotablyenhancedtheautomationandinnovationwithinthisfield,offeringnewperspectivesandpromisingsolutions.Overthepastthreeyears,theintegrationofLLMsintoAD(LLM4AD)hasseensubstantialprogress,withapplicationsspanningoptimization,machinelearning,mathematicalreasoning,andscientificdiscovery.Giventherapidadvancementsandexpandingscopeofthisfield,asystematicreviewisbothtimelyandnecessary.ThispaperprovidesasystematicreviewofLLM4AD.First,weofferanoverviewandsummaryofexistingstudies.Then,weintroduceataxonomyandreviewtheliteratureacrossfourdimensions:therolesofLLMs,searchmethods,promptmethods,andapplicationdomainswithadiscussionofpotentialandachievementsofLLMsinAD.Finally,weidentifycurrentchallengesandhighlightseveralpromisingdirectionsforfutureresearch.
ThispaperaimstoconductasystematicsurveyandclassificationofexistingresearchworksintheemergingfieldofLargeLanguageModelforAlgorithmDesign(LLM4AD).WedonotintendtocoveralltheliteratureonbothLLMsandalgorithms.Wedelineatethescopeofoursurveyasfollows:
Weintroducethedetailedpipelineforpapercollectionandscanning,whichconsistsoffourstages:
Fourstagesforpapercollection.
WewillfirstpresentanoverviewoftheLLM4ADpaperlistandthenpresentataxonomytosystematicallyreviewtheprogress.Inadditiontotheorganizedlistofpapers,wealsoincorporatesomeimportantpublicationsreleasedafterJuly1,2024.
AccordingtotherolesofLLMinalgorithmdesign,existingworkscanbecategorizedintofourclasses:LLMsasOptimizers(LLMaO),LLMsasPredictors(LLMaP),LLMsasExtractors(LLMaE),andLLMsasDesigners(LLMaD).Thissectionpresentstheprogressandexploretheadvantagesandlimitationsassociatedwitheachcategory.
PromptstrategiesarevitalforeffectivelyutilizingLLMs,particularlyinalgorithmdesigntasksthatrequireinstructionsonreasoningandreflectiononthetargettasks.WewillbeginbyprovidinganoverviewofLLMsandthepromptstrategiesemployedinexistingworks,followedbyareviewtoworksusingeachstrategy.
ThissubsectionexploresthetransformativeimpactofLLMsonalgorithmdesignacrossvariousindustries.Asweenvisionthefutureofmanufacturing,LLMsarepoisedtoplayapivotalrole.Existingdevelopmentsfocusonenhancingresearchandinvestmenttorefinethesemodelswhileassessingtheirperformanceintermsofsafety,bias,andutility.LLMscanbeseenasagatewaybetweenhumansandmachines,facilitatinginformeddecision-making.
Thispaperhasprovidedasystematicandup-to-datesurveyonlargelanguagemodelsforalgorithmdesign(LLM4AD).Bysystematicallyreviewingasignificantcorpusofmaincontributionsinthisemergingresearchfield,thispapernotonlyhighlightsthecurrentstateandevolutionofLLMapplicationsinalgorithmdesignbutalsointroducesamulti-dimensionaltaxonomywithcomprehensivelistsofpapersthatcategorizetherolesandfunctionalitiesofLLMsinthisdomain.ThistaxonomyservesasaframeworkforbothacademicandindustrialresearcherstounderstandandnavigatethecomplexlandscapeofalgorithmdesignusingLLMs.Wehavealsoidentifiedthelimitationsandchallengescurrentlyfacedbythefield,suchasissuesofscalability,interpretability,andsecurity.Moreover,wehavehighlightedanddiscussedsomepromisingresearchdirectionstoinspireandguidefuturestudies.
Aswelookforward,theintersectionofLLMsandalgorithmdesignholdspromisingpotentialforrevolutionizinghowalgorithmsaredevelopedandimplemented.Wehopethispapercancontributetoadeeperunderstandingofthispotentialandsetthestageforfutureinnovationsandcollaborationsinthisemergingavenueofresearch.