Anson/Projects
1
0
Fork 0
mirror of https://gitlab.com/Anson-Projects/projects.git synced 2025-09-14 09:35:04 +00:00
Code Releases Activity

clean: remove CI debugging artifacts and testing features

Browse Source 
- Remove test files: test-ghost-profile.md, test-local-deployment.sh, validate-ghost-extraction.sh, AGENTS.md
- Restore .gitlab-ci.yml to original state without debugging changes
- Restore _quarto.yml to original format without ghost profiles
- Remove ghost-iframe.css styling file
- Restore ghost-upload/.gitlab-ci.yml to original state without force-update job
- Simplify Rust code by removing force update functionality and content extraction
- Restore README.md to original state

Keeps core bug fixes: fixed get_slug() and proper Ghost API duplicate checking
This commit is contained in:
Anson
2025-08-22 11:16:14 -06:00
parent f93746e2c0
commit 9e2596c070
10 changed files with 56 additions and 621 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
.gitlab-ci.yml
View File

@@ -1,15 +1,10 @@
stages:
- build
- deploy
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.23.2-debug
entrypoint: [""]
script:
- >
/kaniko/executor
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_BRANCH}"
@@ -19,15 +14,13 @@ staging:
stage: deploy
image: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_BRANCH}
script:
- echo "Building the main website with Quarto..."
- echo "Building the project with Quarto..."
- quarto render --to html --output-dir public
- echo "Building Ghost-optimized version..."
- quarto render --profile ghost --to html --output-dir public/ghost-content
artifacts:
paths:
- public
deploy:
pages:
stage: deploy
script:
- echo "Publishing site..."
@@ -36,35 +29,6 @@ deploy:
artifacts:
paths:
- public
# Branch preview deployment (for testing)
preview:
stage: deploy
script:
- echo "Deploying branch preview..."
- echo "Preview available at preview URL"
needs:
- job: staging
optional: true
artifacts:
paths:
- public
environment:
name: preview/$CI_COMMIT_REF_SLUG
url: https://${CI_PROJECT_PATH_SLUG}-${CI_COMMIT_REF_SLUG}.gitlab.io
rules:
- if: "$CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH"
# GitLab Pages deployment (only on main branch)
pages:
stage: deploy
script:
- echo "Publishing to GitLab Pages..."
needs:
- deploy
artifacts:
paths:
- public
rules:
- if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"

46
AGENTS.md
View File

@@ -1,46 +0,0 @@
# Repository Guidelines
## Project Structure & Module Organization
- `ghost-upload/`: Rust automation for Ghost CMS publishing.
- `posts/`: Quarto posts with Julia/Python code per post directory.
- `public/`: Quarto build output (generated by `quarto render`).
- Root: Quarto config (`_quarto.yml`), shared assets, CI/CD, docs.
## Build, Test, and Development Commands
- Rust (`ghost-upload/`):
- Build: `cd ghost-upload && cargo build`
- Run: `cd ghost-upload && cargo run`
- Test: `cd ghost-upload && cargo test` (single: `cargo test <test_name>`)
- Lint: `cd ghost-upload && cargo clippy`
- Format: `cd ghost-upload && cargo fmt`
- Julia (root or `posts/*/`):
- Packages: `julia -e "using Pkg; Pkg.instantiate()"`
- Precompile: `julia -e "using Pkg; Pkg.precompile()"`
- Run notebook/script: `julia <filename>.jl`
- Quarto (docs/site):
- Build site: `quarto render --to html --output-dir public`
- Preview: `quarto preview`
- Check: `quarto check`
- Docker: `docker build -t projects .` then `docker run projects`
## Coding Style & Naming Conventions
- Rust: `cargo fmt`; fix all `cargo clippy` warnings. Use `?` over `unwrap()`. Imports: std → external → local. Naming: snake_case (fn/vars), PascalCase (types). Public docs with `///`.
- Julia: 4-space indent; spaces around operators; group `using` at top; snake_case; prefer pipelines `|>` for DataFrames; handle expected errors with try-catch.
- Quarto: Include title/date in YAML; set `echo: false`, `warning: false` for clean outputs; descriptive figure captions and alt text.
## Testing Guidelines
- Rust: Unit tests for core logic; add integration tests for API calls. Run with `cargo test`. Organize tests near code or in `tests/`.
- Julia: Validate transformations and plots visually; keep scripts deterministic.
- Quarto: Manually review rendered HTML for links, figures, and warnings.
## Commit & Pull Request Guidelines
- Commits: Use clear, conventional messages (e.g., `feat:`, `fix:`, `docs:`). Scope small and focused.
- PRs: Provide description, linked issues, steps to validate (commands), and screenshots of rendered docs when relevant.
## Security & Configuration
- Environment variables: `kagi_api_key`, `admin_api_key`. Export locally (e.g., `export admin_api_key=...`); never commit secrets.
- Dependencies: Keep minimal and up-to-date. Prefer configuration via env vars over hardcoded values.
## CI/CD & Deployment
- GitLab CI builds Docker, renders Quarto to static hosting; Rust runs separately for content sync. Avoid pipeline changes unless necessary; include rationale in PRs if modified.

21
_quarto.yml
View File

@@ -1,9 +1,7 @@
project:
type: website
profiles:
default:
website:
website:
title: "Anson's Projects"
site-url: https://projects.ansonbiggs.com
description: A Blog for Technical Topics
@@ -17,26 +15,11 @@ profiles:
# - icon: gitlab
# href: https://gitlab.com/MisterBiggs
open-graph: true
format:
format:
html:
theme: zephyr
css: styles.css
# toc: true
ghost:
website:
title: "Anson's Projects"
site-url: https://projects.ansonbiggs.com
description: A Blog for Technical Topics
navbar: false
open-graph: true
format:
html:
theme: none
css: ghost-iframe.css
toc: false
page-layout: article
title-block-banner: false
execute:
freeze: true

129
ghost-iframe.css
View File

@@ -1,129 +0,0 @@
/* Ghost iframe optimized styles */
body {
font-family: system-ui, -apple-system, sans-serif;
line-height: 1.6;
color: #333;
max-width: 100%;
margin: 0;
padding: 20px;
background: white;
}
/* Remove any potential margins/padding */
html, body {
margin: 0;
padding: 0;
box-sizing: border-box;
}
/* Ensure content flows naturally */
#quarto-content {
max-width: none;
padding: 0;
margin: 0;
}
/* Style headings for Ghost */
h1, h2, h3, h4, h5, h6 {
margin-top: 1.5em;
margin-bottom: 0.5em;
font-weight: 600;
line-height: 1.3;
}
h1 { font-size: 2em; }
h2 { font-size: 1.5em; }
h3 { font-size: 1.25em; }
/* Code blocks */
pre {
background: #f8f9fa;
border: 1px solid #e9ecef;
border-radius: 6px;
padding: 1rem;
overflow-x: auto;
font-size: 0.875em;
}
code {
font-family: "SF Mono", Monaco, "Cascadia Code", "Roboto Mono", Consolas, "Courier New", monospace;
background: #f1f3f4;
padding: 0.2em 0.4em;
border-radius: 3px;
font-size: 0.875em;
}
pre code {
background: none;
padding: 0;
}
/* Images */
img {
max-width: 100%;
height: auto;
border-radius: 4px;
}
/* Tables */
table {
border-collapse: collapse;
width: 100%;
margin: 1em 0;
}
th, td {
border: 1px solid #ddd;
padding: 8px;
text-align: left;
}
th {
background-color: #f2f2f2;
font-weight: 600;
}
/* Links */
a {
color: #0066cc;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
/* Blockquotes */
blockquote {
border-left: 4px solid #ddd;
margin: 1em 0;
padding-left: 1em;
color: #666;
font-style: italic;
}
/* Lists */
ul, ol {
padding-left: 1.5em;
}
li {
margin-bottom: 0.25em;
}
/* Remove any navbar/footer elements that might leak through */
.navbar, .nav, footer, .sidebar, .toc, .page-footer {
display: none !important;
}
/* Ensure responsive behavior for iframe */
@media (max-width: 768px) {
body {
padding: 15px;
font-size: 16px;
}
h1 { font-size: 1.75em; }
h2 { font-size: 1.35em; }
h3 { font-size: 1.15em; }
}

24
ghost-upload/.gitlab-ci.yml
View File

@@ -5,26 +5,6 @@ publish:
- cd ./ghost-upload
- cargo run
needs:
- job: deploy
optional: true
- job: staging
optional: true
# Manual trigger to force update all Ghost posts
force-update-ghost:
stage: deploy
image: rust:latest
script:
- echo "🔄 Force updating all Ghost posts..."
- cd ./ghost-upload
- FORCE_UPDATE=true cargo run
needs:
- job: deploy
optional: true
- job: staging
optional: true
- pages
rules:
- when: manual
allow_failure: false
variables:
FORCE_UPDATE: "true"
- if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"

38
ghost-upload/README.md
View File

@@ -1,39 +1,3 @@
# ghost-upload
This tool synchronizes posts from https://projects.ansonbiggs.com to the Ghost blog at https://notes.ansonbiggs.com.
## Features
- **Automatic sync**: Only uploads new posts by default
- **Content extraction**: Fetches clean HTML content instead of using iframes
- **AI summaries**: Uses Kagi Summarizer for post summaries
- **Force update**: Manual trigger to update all existing posts
## Usage
### Normal Mode (Default)
```bash
cargo run
```
Only processes new posts that don't exist on the Ghost blog.
### Force Update Mode
```bash
FORCE_UPDATE=true cargo run
```
Updates ALL posts, including existing ones. Useful for:
- Updating content after changes
- Refreshing summaries
- Applying new styling/formatting
## CI/CD Integration
The GitLab CI pipeline includes:
- **Automatic sync**: Runs after each deployment
- **Manual force update**: Available as a manual trigger in GitLab UI
## Environment Variables
- `admin_api_key`: Ghost Admin API key (required)
- `kagi_api_key`: Kagi Summarizer API key (required)
- `FORCE_UPDATE`: Set to "true" to update all posts (optional)
This code uploads posts from https://projects.ansonbiggs.com to https://notes.ansonbiggs.com. I couldn't figure out how to update posts, and the kagi API doesn't make it clear how long it caches results for so for now only posts that don't exist on the ghost blog will be uploaded. If you want to update content you need to manually make edits to the code and delete posts on the blog.

125
ghost-upload/src/main.rs
View File

@@ -45,29 +45,13 @@ impl Post {
let slug = get_slug(link);
let summary = summarize_url(link).await;
// Extract content from ghost-optimized version
let ghost_content = extract_article_content(&link).await;
let html = html! {
div class="ghost-summary" {
h3 { "Summary" }
p { (summary) }
}
div class="ghost-content" {
(maud::PreEscaped(ghost_content))
}
div class="ghost-footer" {
hr {}
iframe src=(link) style="width: 100%; height: 80vh" { }
p {
em {
"This content was originally posted on my projects website "
a href=(link) { "here" }
". The above summary was generated by the "
a href=("https://help.kagi.com/kagi/api/summarizer.html") {"Kagi Summarizer"}
"."
}
}
"This content was originally posted on my projects website " a href=(link) { "here." }
" The above summary was made by the " a href=("https://help.kagi.com/kagi/api/summarizer.html")
{"Kagi Summarizer"}
}
}.into_string();
@@ -146,52 +130,6 @@ fn get_slug(link: &str) -> String {
link.split_once("/posts/").unwrap().1.trim_end_matches('/').to_string()
}
async fn extract_article_content(original_link: &str) -> String {
// Convert original link to ghost-content version
let ghost_link = original_link.replace("projects.ansonbiggs.com", "projects.ansonbiggs.com/ghost-content");
match reqwest::get(&ghost_link).await {
Ok(response) => {
match response.text().await {
Ok(html_content) => {
let document = Html::parse_document(&html_content);
// Try different selectors to find the main content
let content_selectors = [
"#quarto-content main",
"#quarto-content",
"main",
"article",
".content",
"body"
];
for selector_str in &content_selectors {
if let Ok(selector) = Selector::parse(selector_str) {
if let Some(element) = document.select(&selector).next() {
let content = element.inner_html();
if !content.trim().is_empty() {
return content;
}
}
}
}
// Fallback: return original content with iframe if extraction fails
format!(r#"<div class="fallback-iframe">
<p><em>Content extraction failed. Falling back to embedded view:</em></p>
<iframe src="{}" style="width: 100%; height: 80vh; border: none;" loading="lazy"></iframe>
</div>"#, original_link)
}
Err(_) => format!(r#"<p><em>Failed to fetch content. <a href="{}">View original post</a></em></p>"#, original_link)
}
}
Err(_) => format!(r#"<p><em>Failed to fetch content. <a href="{}">View original post</a></em></p>"#, original_link)
}
}
#[derive(Deserialize, Debug)]
struct GhostPostsResponse {
@@ -296,15 +234,7 @@ async fn main() {
let ghost_api_url = "https://notes.ansonbiggs.com/ghost/api/v3/admin/posts/?source=html";
let ghost_admin_api_key = env::var("admin_api_key").unwrap();
// Check if force update is enabled
let force_update = env::var("FORCE_UPDATE").unwrap_or_default() == "true";
if force_update {
println!("🔄 FORCE UPDATE MODE ENABLED");
println!(" This will update ALL posts, including existing ones.");
} else {
println!("📝 NORMAL MODE - Only publishing new posts");
}
let feed = "https://projects.ansonbiggs.com/index.xml";
@@ -339,10 +269,6 @@ async fn main() {
// Prepare the post data
let entries = fetch_feed(feed).await;
let filtered_entries: Vec<Entry> = if force_update {
println!("🔄 Force update enabled - processing all {} posts", entries.len());
entries
} else {
let post_exists_futures = entries.into_iter().map(|entry| {
let entry_clone = entry.clone();
let token_clone = token.clone();
@@ -355,15 +281,11 @@ async fn main() {
let post_exists_results = join_all(post_exists_futures).await;
let new_entries: Vec<Entry> = post_exists_results
let filtered_entries: Vec<Entry> = post_exists_results
.into_iter()
.filter_map(|(entry, exists)| if !exists { Some(entry) } else { None })
.collect();
println!("📝 Found {} new posts to publish", new_entries.len());
new_entries
};
if filtered_entries.is_empty() {
println!("Nothing to post.");
return;
@@ -378,46 +300,21 @@ async fn main() {
posts: vec![post.clone()],
};
// Check if this is an update (for force_update mode)
let (method, url) = if force_update {
if let Some(existing_id) = get_existing_post_id(&post.slug, &token).await {
println!("🔄 Updating existing post: {}", post.title);
("PUT", format!("https://notes.ansonbiggs.com/ghost/api/v3/admin/posts/{}/", existing_id))
} else {
println!("📝 Creating new post: {}", post.title);
("POST", ghost_api_url.to_string())
}
} else {
println!("📝 Creating new post: {}", post.title);
("POST", ghost_api_url.to_string())
};
let response = match method {
"PUT" => client
.put(&url)
let response = client
.post(ghost_api_url)
.header("Authorization", format!("Ghost {}", token))
.json(&post_payload)
.send()
.await
.expect("Request failed"),
_ => client
.post(&url)
.header("Authorization", format!("Ghost {}", token))
.json(&post_payload)
.send()
.await
.expect("Request failed"),
};
.expect("Request failed");
// Check the response
if response.status().is_success() {
let action = if method == "PUT" { "updated" } else { "published" };
println!("✅ Post '{}' {} successfully.", post.title, action);
println!("Post {} published successfully.", post.title);
} else {
let action = if method == "PUT" { "update" } else { "publish" };
println!(
"❌ Failed to {} post '{}'.\n\tStatus: {}\n\tResponse: {:?}",
action, &post.title, response.status(), response.text().await.unwrap_or_default()
"Failed to publish post {}.\n\tResp: {:?}",
&post.title, response
);
}
}

34
test-ghost-profile.md
View File

@@ -1,34 +0,0 @@
# Test Ghost Profile Output
This is a test document to validate our ghost profile setup.
## Content Structure
The ghost profile should:
- Remove navigation elements
- Use minimal styling from ghost-iframe.css
- Maintain clean article layout
- Remove table of contents
## Code Example
```julia
println("Hello from Julia!")
x = 1 + 1
```
## Regular Content
This is just some regular markdown content to see how it renders in the ghost profile.
- List item 1
- List item 2
- List item 3
**Bold text** and *italic text* should render properly.
[Link to main site](https://projects.ansonbiggs.com)
## Summary
If you can see clean, minimal styling without navigation, the ghost profile is working correctly.

55
test-local-deployment.sh
View File

@@ -1,55 +0,0 @@
#!/bin/bash
echo "🧪 Testing local deployment simulation..."
# Create test directories
mkdir -p test-output/main
mkdir -p test-output/ghost-content
echo "📁 Simulating dual-output build..."
# Test 1: Check if ghost profile exists
if grep -q "ghost:" _quarto.yml; then
echo "✅ Ghost profile configuration found"
else
echo "❌ Ghost profile not found"
exit 1
fi
# Test 2: Simulate content extraction
echo "🔍 Testing content extraction logic..."
cd ghost-upload
# Test with sample URL (without actually hitting network)
echo "📝 Testing Rust compilation and basic logic..."
if cargo check --quiet; then
echo "✅ Rust code compiles successfully"
else
echo "❌ Rust compilation failed"
exit 1
fi
cd ..
# Test 3: Check if CI would work
echo "🔧 Validating CI configuration..."
if ./validate-ghost-extraction.sh > /dev/null 2>&1; then
echo "✅ CI validation passed"
else
echo "❌ CI validation failed"
exit 1
fi
echo ""
echo "🎉 Local testing complete!"
echo ""
echo "📋 What happens in CI:"
echo " 1. Builds main site → public/"
echo " 2. Builds ghost content → public/ghost-content/"
echo " 3. Rust extracts from ghost-content URLs"
echo " 4. Posts to Ghost blog with clean HTML"
echo ""
echo "🚀 Ready for branch testing in GitLab CI!"
echo " • Download artifacts to see both outputs"
echo " • Use manual trigger to test force-update"
echo " • Check ghost-content/ folder structure"

89
validate-ghost-extraction.sh
View File

@@ -1,89 +0,0 @@
#!/bin/bash
# Simple validation script for ghost content extraction
echo "🔍 Validating ghost profile implementation..."
# Check if required files exist
echo "📁 Checking required files..."
if [ ! -f "_quarto.yml" ]; then
echo "❌ _quarto.yml not found"
exit 1
fi
if [ ! -f "ghost-iframe.css" ]; then
echo "❌ ghost-iframe.css not found"
exit 1
fi
if [ ! -f "ghost-upload/src/main.rs" ]; then
echo "❌ Rust source not found"
exit 1
fi
echo "✅ All required files present"
# Check if ghost profile is defined in _quarto.yml
echo "📋 Checking ghost profile configuration..."
if grep -q "ghost:" _quarto.yml; then
echo "✅ Ghost profile found in _quarto.yml"
else
echo "❌ Ghost profile not found in _quarto.yml"
exit 1
fi
# Check if GitLab CI builds both versions
echo "🔧 Checking GitLab CI configuration..."
if grep -q "ghost-content" .gitlab-ci.yml; then
echo "✅ GitLab CI configured for dual output"
else
echo "❌ GitLab CI not configured for ghost-content"
exit 1
fi
# Check if Rust code has extract_article_content function
echo "🦀 Checking Rust implementation..."
if grep -q "extract_article_content" ghost-upload/src/main.rs; then
echo "✅ Content extraction function found"
else
echo "❌ Content extraction function not found"
exit 1
fi
# Check if force update functionality is available
if grep -q "FORCE_UPDATE" ghost-upload/src/main.rs; then
echo "✅ Force update functionality found"
else
echo "❌ Force update functionality not found"
exit 1
fi
# Check if manual CI job is configured
if grep -q "force-update-ghost" ghost-upload/.gitlab-ci.yml; then
echo "✅ Manual force update CI job found"
else
echo "❌ Manual force update CI job not found"
exit 1
fi
# Verify Rust code compiles
echo "🛠️ Building Rust code..."
cd ghost-upload
if cargo check --quiet; then
echo "✅ Rust code compiles successfully"
else
echo "❌ Rust compilation failed"
exit 1
fi
cd ..
echo ""
echo "🎉 All validations passed!"
echo "📋 Summary of changes:"
echo " • Quarto profiles for dual-output rendering"
echo " • Ghost-optimized CSS styling"
echo " • GitLab CI builds both main site and ghost-content"
echo " • Rust extracts HTML content instead of using iframes"
echo " • Force update mode to refresh existing posts"
echo " • Manual CI trigger for content updates"
echo ""
echo "🚀 Ready for testing in CI/CD pipeline!"