---
title: "Scraping web articles for AI"
description: "Scraping web articles has never been this easy. Heres some tools to make it even easier and help feed your AI more data."
author: "wompmacho"
date: '2026-05-17T12:15:13-04:00'
lastmod: '2026-05-17'
tags: ["golang", "automation", "markdown", "gemini-cli", "skills"]
---

## Introduction

Managing knowledge often involves capturing articles and documentation from the web. To streamline this workflow, I developed `web-to-markdown` as a specialized utility written in Go that extracts article content, converts it to clean Markdown, and downloads inline images locally (*structured to compliment my hugo site*).

This is really good for quickly grabbing / sanitizing data and providing a great deal of context for agents. Adding this tool as a skill really compliments the planning stage of a project. 

## The web-to-markdown Utility

The core utility (*[git repos](https://git.wompmacho.com/wompmacho/web-to-markdown)*) is a highly optimized CLI application built with Go `1.25.0+`. It is designed to be fast, reliable, and to produce a clean output structure.

### Key Features

- **Boilerplate Removal:** The tool leverages the `go-readability` library to intelligently isolate the main article content, automatically stripping out distracting elements such as advertisements, navigation bars, and footers.  
- **Concurrent Image Downloading:** Performance is maximized by utilizing native Goroutines to download all inline images simultaneously. This significantly reduces the time required to process image-heavy articles.  
- **Markdown Conversion:** The sanitized HTML is converted into readable Markdown using the `html-to-markdown` package.  
- **Intelligent Output Structure:** The utility generates a flat directory structure. It saves the main article as `index.md` and intelligently renames all downloaded images based on their `alt` text, `<figcaption>`, or surrounding header context. The markdown links are automatically rewritten to point to these local, contextualized image names.

### Usage Example

The utility is executed via the command line, accepting the target URL and optional flags for customization.

```sh
./web-to-markdown -title "Go Concurrency Guide" -out "./docs/go" "https://example.com/go-concurrency"
```

This command generates a structured output similar to the following:

```sh
docs/go/
└── go-concurrency-guide/
    ├── index.md
    ├── diagram-of-goroutines-a1b2c3.jpg
    └── author-profile-pic-f6g7h8.png
```

## Extending the Agent with a Custom Skill

While the CLI tool is powerful on its own, manually executing it interrupts the creative or research flow. To solve this, we developed a custom Gemini CLI skill.

Skills in Gemini CLI are modular packages that inject specialized procedural knowledge and workflows into the agent's context window.

### The web-to-markdown Skill

The custom skill we created instructs the Gemini CLI agent on how and when to use the local `web-to-markdown` binary.

When a user issues a command like, *"Grab me a copy of `https://example.com/article`,"* the skill triggers the following automated workflow:

1. **Configuration Gathering:** The agent pauses and asks the user where the article should be saved and what the desired title should be.  
2. **Execution:** Once the parameters are confirmed, the agent autonomously executes the `run_shell_command` tool, invoking the `web-to-markdown` utility with the correct flags and URL.  
3. **Verification:** The agent verifies the success of the command by checking the target directory and informs the user that the Markdown file is ready for review.

### Skill Creation Process

The skill was generated using the built-in `skill-creator`. The process involved:

1. **Initialization:** Running the initialization script to scaffold the skill directory structure.  
2. **Drafting Instructions:** Writing the `SKILL.md` file, which includes the YAML frontmatter (defining the trigger description) and the step-by-step workflow instructions for the agent.  
3. **Packaging and Installation:** Compiling the skill into a `.skill` archive and installing it into the user's `~/.gemini/skills/` directory.