nice-gee/scripts
kbanc85 9956c9dee4 Improve extraction scripts, validation, and documentation
Enhance article extraction and validation systems with better error handling, retry logic, and content cleaning.

Script improvements:
- Add retry logic with exponential backoff for API calls
- Implement 60-second timeout for Anthropic API requests
- Improve HTML cleaning with Substack-specific filters
- Remove navigation, footer, and UI noise from extracted content
- Better HTML entity decoding

Validation enhancements:
- Add duplicate URL detection across articles
- Validate topic assignments against known topics
- Enhanced error reporting with specific issue types

GitHub Actions:
- Add build output validation step
- Verify /out directory exists before PR creation
- Count generated HTML files for sanity check

Documentation updates:
- Update project stats (26 articles, ~30 pages)
- Document v1.2 improvements
- Add changelog entries

Type safety:
- Improve topics.ts type definitions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 14:06:19 -05:00
..
add-claim-to-data.ts Fix comma insertion logic to prevent undefined entries 2025-11-06 11:46:32 -05:00
audit-claims.ts Clean up claims data and add validation system 2025-11-06 17:54:07 -05:00
auto-add-new-articles.ts Add automated article processing system 2025-11-05 21:31:07 -05:00
check-new-articles.ts Remove Office Hour entry and filter live broadcasts 2025-11-06 17:58:16 -05:00
comprehensive-claim-fixer.ts Clean up claims data and add validation system 2025-11-06 17:54:07 -05:00
extract-article-data-old.ts.bak Implement two-tier AI extraction for 19% cost savings 2025-11-06 13:10:22 -05:00
extract-article-data.ts Improve extraction scripts, validation, and documentation 2025-11-09 14:06:19 -05:00
find-duplicates.ts Clean up claims data and add validation system 2025-11-06 17:54:07 -05:00
find-similar-articles.ts Clean up claims data and add validation system 2025-11-06 17:54:07 -05:00
fix-all-claims.ts Clean up claims data and add validation system 2025-11-06 17:54:07 -05:00
generate-claim-fixes.ts Clean up claims data and add validation system 2025-11-06 17:54:07 -05:00
generate-rss.ts Add defensive checks to RSS/sitemap generation 2025-11-06 11:16:57 -05:00
generate-sitemap.ts Add defensive checks to RSS/sitemap generation 2025-11-06 11:16:57 -05:00
merge-migrated-data.ts Add 2 new claims pages using automated AI extraction 2025-11-04 14:29:42 -05:00
remove-duplicates.ts Clean up claims data and add validation system 2025-11-06 17:54:07 -05:00
validate-claims.ts Improve extraction scripts, validation, and documentation 2025-11-09 14:06:19 -05:00