{"data":{"event":{"id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","slug":"from-prompt-to-pipeline-a-comparative-evaluation-of-large-language-model-02c05d25f4","title":"\n\nFrom Prompt to Pipeline: A Comparative Evaluation of Large Language Model Coding Agents for Reproducible Bioinformatics Pipeline Construction \n\n","short_summary":"Agentic coding systems are increasingly presented as a way to reduce the engineering burden of scientific software development. Bioinformatics is a strong test case for this claim because useful pipelines must combine domain-specific analysis choices, command-line software, sample metadata, workflow orchestration, container or HPC execution, and interpretable quality-control reporting. We evaluated three agentic systems - Biomni, Claude Code, and Codex - on the same task: constructing a Nextflow DSL2 pipeline for paired-end CUT&Tag data that included read QC, trimming, alignment, filtering, du","full_description":"Agentic coding systems are increasingly presented as a way to reduce the engineering burden of scientific software development. Bioinformatics is a strong test case for this claim because useful pipelines must combine domain-specific analysis choices, command-line software, sample metadata, workflow orchestration, container or HPC execution, and interpretable quality-control reporting. We evaluated three agentic systems - Biomni, Claude Code, and Codex - on the same task: constructing a Nextflow DSL2 pipeline for paired-end CUT&Tag data that included read QC, trimming, alignment, filtering, duplicate removal, signal track generation, per-sample and group-level peak calling, control-aware group merging, annotation, FRiP calculation, deepTools visualizations, and final MultiQC reporting. Each system received the same detailed CRAFT-style prompt and was assessed against a hand-coded reference pipeline developed by the authors. All three systems produced pipeline implementations that appeared plausible at the level of documentation and file structure, but none fully satisfied the requested analysis. The most consequential failure was shared: the agent-generated pipelines performed some form of group-level merging but did not produce the requested merged-group reporting outputs. Sample-level MultiQC reports also disagreed with the reference report. Codex was closest to the reference for primary mapped-read counts, although its total-read accounting and report structure still differed. Claude Code produced the broadest final report, but its mapping summary mixed stages and therefore could not be treated as numerically correct. Biomni produced the strongest subjective documentation, but its read-count agreement with the reference report was poor and several failures required substantial Nextflow expertise to diagnose. These results suggest that current coding agents can accelerate scaffolding, documentation, and routine implementation, but they do not eliminate the need for expert review in bioinformatics workflow construction. For complex sequencing workflows, prompts must specify not only the biological intent, but also the exact stage semantics, acceptance tests, metadata contracts, expected report sections, resource propagation rules, and failure criteria needed to distinguish a plausible pipeline from a correct one.\n\nUpdate (2026-09-14): Show HN: Kinesis – Control your Mac with the Meta Neural Band. a couple hours ago i had astra take on a 10 month old repo of mine https://github.com/callbacked/neural-band-poc/ that involved finding a way to use the meta neural band independently from the glasses to read the sEMG data off of it. i really think it is a cool piece of technology that largely gets overlooked by the glasses, and always thought it was a shame it couldn't be used outside of its main application.\nafter making quick work of that, i wanted to use it to control my mac using gesture controls and that's how kinesis was born. it has been very cool being able to swipe from desktops using my fingers, swiping up to open up mission control or controlling a virtual knob to adjust my volume.\ni definitely havent scratched the surface with the neural band but it has been a really fun endeavor thus far! i really hope others can find creative uses for their neural bands too :)\nnote: i really hope this works on other bands because the sample size has just been me so far.\nComments URL: https://news.ycombinator.com/item?id=49695408\nPoints: 64\n# Comments: 20","ledger_type":"benefit","primary_domain_id":"8f1af1b9-7302-4b51-aae9-fdfac04d158a","event_status":"provisional","event_date":"2026-09-17T00:00:00.000Z","discovery_date":"2026-09-17T00:00:00.000Z","first_published_date":"2026-09-17T00:00:00.000Z","last_reviewed_date":"2026-09-14T00:00:00.000Z","geographic_scope":"International","affected_population":null,"base_impact_tier":1,"base_score":"1.00","attribution_multiplier":"0.1000","evidence_multiplier":"0.2000","realization_multiplier":"0.2000","durability_multiplier":"0.5000","current_event_score":"0.002000","confidence_level":"low","score_explanation":"Ingest update: independent source raised evidence; added a follow-up source. Score remains provisional until review.","methodology_version_id":"8745541b-0f8e-4d5a-9b47-5291d3813a8d","original_methodology_version_id":"8745541b-0f8e-4d5a-9b47-5291d3813a8d","published_at":"2026-09-18T03:00:50.854Z","created_at":"2026-09-18T03:00:50.854Z","updated_at":"2026-09-18T04:00:48.081Z","flags":[],"domain_name":"Biology","domain_slug":"biology","methodology_version":"0.2"},"contributions":[{"id":"c128e7b4-00c5-495c-83d3-dfca541ed9c2","event_id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","model_id":"d96eca85-24db-4c1d-9a28-ca66dc5a2847","role_description":"Named in the ingested source. Credit is provisional until review.","attribution_multiplier":"0.1000","credit_share":"0.5000","contribution_score":"0.001000","attribution_rationale":"Ingest matched a model alias in the source text. Attribution stays provisional.","attribution_confidence":"medium","first_used_date":"2026-09-17T00:00:00.000Z","model_version_if_known":null,"review_status":"approved","created_at":"2026-09-18T03:00:50.877Z","model_slug":"openai-codex","model_name":"OpenAI Codex","identity_class":"named_release","is_internal":false,"family_name":"Codex","family_slug":"openai-codex","organization_name":"OpenAI","organization_slug":"openai"},{"id":"cf5b820f-88d9-4679-b543-d1dd03260fe8","event_id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","model_id":"570f77f8-5233-4c57-9ce5-7753b481beee","role_description":"Named in the ingested source. Credit is provisional until review.","attribution_multiplier":"0.1000","credit_share":"0.5000","contribution_score":"0.001000","attribution_rationale":"Ingest matched a model alias in the source text. Attribution stays provisional.","attribution_confidence":"medium","first_used_date":"2026-09-17T00:00:00.000Z","model_version_if_known":null,"review_status":"approved","created_at":"2026-09-18T03:00:50.870Z","model_slug":"claude-3-7-sonnet","model_name":"Claude 3.7 Sonnet","identity_class":"named_release","is_internal":false,"family_name":"Claude","family_slug":"anthropic-claude","organization_name":"Anthropic","organization_slug":"anthropic"}],"sources":[{"id":"3ae8034f-d208-42ce-890b-a45a390188cf","event_id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","url":"\nhttps://www.biorxiv.org/content/10.64898/2026.09.11.751004v1?rss=1\n","canonical_url":"\nhttps://www.biorxiv.org/content/10.64898/2026.09.11.751004v1?rss=1\n","source_type":"preprint","publisher":"bioRxiv","author":null,"publication_date":"2026-09-17T00:00:00.000Z","retrieved_at":"2026-09-18T03:00:50.886Z","title":"\n\nFrom Prompt to Pipeline: A Comparative Evaluation of Large Language Model Coding Agents for Reproducible Bioinformatics Pipeline Construction \n\n","excerpt":"Agentic coding systems are increasingly presented as a way to reduce the engineering burden of scientific software development. Bioinformatics is a strong test case for this claim because useful pipelines must combine domain-specific analysis choices, command-line software, sample metadata, workflow orchestration, container or HPC execution, and interpretable quality-control reporting. We evaluated three agentic systems - Biomni, Claude Code, and Codex - on the same task: constructing a Nextflow","content_hash":null,"source_reliability_class":"medium","is_primary_source":true,"is_independent":true,"is_peer_reviewed":false,"archived_url":null,"created_at":"2026-09-18T03:00:50.886Z"},{"id":"f7d57e79-f8c3-454c-9d5d-c2434882fe62","event_id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","url":"https://github.com/callbacked/kinesis","canonical_url":"https://github.com/callbacked/kinesis","source_type":"journalism","publisher":"Hacker News","author":null,"publication_date":"2026-09-14T00:00:00.000Z","retrieved_at":"2026-09-18T04:00:48.048Z","title":"Show HN: Kinesis – Control your Mac with the Meta Neural Band","excerpt":"a couple hours ago i had astra take on a 10 month old repo of mine https://github.com/callbacked/neural-band-poc/ that involved finding a way to use the meta neural band independently from the glasses to read the sEMG data off of it. i really think it is a cool piece of technology that largely gets overlooked by the glasses, and always thought it was a shame it couldn't be used outside of its main application.\nafter making quick work of that, i wanted to use it to control my mac using gesture co","content_hash":null,"source_reliability_class":"medium","is_primary_source":false,"is_independent":true,"is_peer_reviewed":false,"archived_url":null,"created_at":"2026-09-18T04:00:48.048Z"}],"claims":[{"id":"595f79ad-78da-47c5-82e3-58372ca21ac6","event_id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","claim_text":"\n\nFrom Prompt to Pipeline: A Comparative Evaluation of Large Language Model Coding Agents for Reproducible Bioinformatics Pipeline Construction \n\n","claim_type":"outcome","claim_status":"supported","confidence_score":null,"created_at":"2026-09-18T03:00:50.894Z","updated_at":"2026-09-18T03:00:50.894Z"}],"revisions":[{"id":"bbad2d3e-cdb1-4f45-a6b8-7c5db50e82d2","event_id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","model_id":null,"previous_score":"0.000000","new_score":"0.001000","previous_factors":{},"new_factors":{"tier":1,"evidence":0.1,"base_score":1,"durability":0.5,"attribution":0.1,"realization":0.2},"change_reason":"Auto-published from news ingest.","trigger_type":"news_ingest","trigger_source_ids":null,"reviewer_id":null,"review_status":"published","created_at":"2026-09-18T03:00:50.903Z"},{"id":"31c6048b-8e15-41eb-9df0-40cad35a3587","event_id":"bf4aec04-677b-4eb6-a7d7-e5ddcc55a620","model_id":null,"previous_score":"0.001000","new_score":"0.002000","previous_factors":{"tier":1,"evidence":0.1,"base_score":1,"durability":0.5,"attribution":0.1,"realization":0.2},"new_factors":{"tier":1,"models":["GPT-6 Astra"],"evidence":0.2,"base_score":1,"durability":0.5,"attribution":0.1,"realization":0.2},"change_reason":"Ingest update: independent source raised evidence; added a follow-up source.","trigger_type":"ingest_update","trigger_source_ids":["f7d57e79-f8c3-454c-9d5d-c2434882fe62"],"reviewer_id":null,"review_status":"published","created_at":"2026-09-18T04:00:48.088Z"}],"secondary":[]},"methodology_version":"0.1"}