Updates

Why aren't etags working properly?

Tell Chuck that photo posts are dated at the time they appear in Blot. I'll consider adding an option to use the image taken date?

Google Drive blockquotes and tell Chuck

Investigate iCloud bugs

• Repeat issue with files in sub folders...
• What happens when you rename a folder connected to iCloud client?
  • does this explain why there are unrenamed folders in the mac server icloud drive?
• Fix subfolder sync issue with iCloud images and tell James and Shawn and Cameron
• Download Haikun's folder and fix issue with iCloud
• Work out issue with iCloud disconnect / setup with Florian's folder
• remove all keys related to icloud client for blog_e74c2a93cXXXX

Fix bug with navigation on blog template on mobile

Improve folder to CDN pipeline

• Could we respond with redirects to Blot's CDN?
• Why isn't the WAV file in this footnote being properly replaced with a CDN URL?
  • it's because it contains URL-encoded chars – replace
  • https://diwank.space/field-notes-from-shipping-real-code-with-claude
• Why isn't this file?
  • https://yangrobotics.com/posts/peng-2.md
  • possibly because it's a fully-qualified URL? we need to handle these I think

Get a handle on data egress from AWS – block bots?

• Check in on this chart

Fix issue with template settings and tell Dirt

Add per request resource monitoring, e.g. heap size before/after

Tune proxy next upstream status code conditions to preserve zero-downtime-deployments but prevent expensive requests (timeout?)

Fix various performance issues with archives page and sitemap

• highlighted by https://salocin.blot.im and https://salocin.blot.im/archives

Implement mac server screenshot endpoint

• Remove weird fake screenshot code from template editor

Implement limit for allentries, sitemap, etc.

Set long-timeout for git client requests on reverse proxy

• prevent request retry for non

Fix bug with Bruce's local template, possibly related to Ryan's issue. The template switches from 3 to 2 after each sync.

• Also affecting Chris

Fix accessibility issue with menu links reordering and tell Grace

Investigate why thumbnail generator is failing for posts on Chris' site

• it's to do with 304 responses in transformer - handle these properly
• fix issue with 304s for Fabrice e.g. on this post: https://www.poirpom.com/bas-les-cloches

Add tests for webp thumbnails

Add deploy button to GitHub so I can deploy code while away

---

Add a related entry list and tell David and Thomas and questioner and questioner

Work out issue with initial transfer and Michael's folder

Move folder lock location to instance ssd for improved reliability and performance (non-networked drive...) – could be inside the cache

---

Improve replaceFolderLinks to handle situation where people include their own domain in image links –> parse this out

• fix issue with discombobulated

Run dropbox reset as ec2-user not root

• run chown on all data files...

Add blotignore feature and tell Troy

Are there any missing templates because of the switch from lowercase to true case?

Fix issues with menu links based on case-sensitivity switch

Fix issues with bad permalinks based on case-sensitivity switch

Fix template partial embeddings which no longer work because of switch from lowercase to true case:

• e.g. {{> /pages/home.txt}} when pages is really Pages
• we should resolve the case-insensitive path to file when saving a template and parsing its partials

How to fix menu links with incorrect case IDs?

• Write script to catch these and patch

Why do posts on blog_eddd8e5 take so long to set, let alone rebuild?

Clean out all but the neccessary information from deleted entries to save redis space

We should use the helper/transformer to speed up screenshot re-generation

Watch the logs for requests to files in blog folder which aren't passed to CDN

• add some logging to blog/assets.js

Make iCloud site routes blocking so the mac server waits for the main server to finish building before sending the next sync update? But consider timeout

Think about how to purge the old versions of files cached by bunny's permacache

Fix bug which prevents the CDN url replacer working for drafts

• Refactor so the draft route relies more on the middleware function
• At the moment, drafts bypass a lot of the code – many template features won't work

How can we serve template files over CDN?

• do we have to remap references to other files in stylesheets since we're doing a switcheroo

Image cacher should work for webp, tiff, etc. as well as JPG

Fix bug with template error causing server to go down (issue with folder rebuild) reproduce bug with chinese template?

Fix bug with search for 'Raycast' and tell Anders

Add script to resync all icloud folders – make sure they are up to date

Check that Google Drive is staying in sync:

• login -> node scripts/google-drive/reset.js
• add proper bottlenecks then run this regularly

Come up with plan to handle google docs on disconnection from Google drive -> convert to markdown?

• tell henryirvine about issues with his site

Write proper tests for clients:

• Dropbox
• Google Drive
• iCloud

Make initial health check /health more powerful

• inspect blogs directory and verify there are the expected number of folders
• check we can communicate with redis
• verify pandoc is installed and running
• make the initial timeout in the health check longer?
• subsequent requests can just check redis connection?

Use sockets for communication between nginx and node

Why don't empty files sync on the dashboard? (esp. via iCloud client)

Improve speed of production build

• When should folder zipping happen?
• Can we improve build speed by removing github actions cache and using some base image with dependencies?

Migrate scripts in bashrc into deployment flow

• run them all inside backup server -blot-blue

Use npm run test in production

• echo github env -> scripts/tests/.env

---

Convert callbacks to async/await

• literally anywhere in the codebase – use util.callbackify to add backwards compatibility

Try and reorganize code into a cleaner MVC division. It's really the controllers that need to be moved.

Write unit tests for database models which have no tests subdirectory

• e.g. models/404, models/entries, models/redirects

Improve sidebar blog template navigation on small screens

Improve screenshot utility:

• handle / detect pupeteer browser crash and restart
• add global timeout top-level screenshot function for individual invocations
• improve fonts if possible?
• the source code for the screenshot is in helper/screenshot and it's only used in app/build/
• improve and extend tests for resilience, buggy webpages – verify 100% test coverage
• Log sites which cause screenshot function to error or timeout
  • sapiens.org
  • harpers.org

Clamp length of password and email address in log in and sign up and reset forms

• ensure the forms have tight max-bodies
• return an error if these are too long

---

Speed up templates.build

• It seems setView already checks for changes
• Try and be more selective about which blog model updates trigger blog.flushCache
• Write now sync updates (e.g. checking /) all trigger blogs to flush cache

Move data dir to s3

• Come up with plan for static files directory
  • abstract write / read from static files directory
• Come up with plan for blog files directory
• abstract write / read from blog directory
• If we can come up with a git client plan that works, this is a go
  • Git S3: https://news.ycombinator.com/item?id=41887004

Fix bad links (e.g. incorrect case) in files at build-time rather than run-time so we can avoid the expensive disk lookups in blog/assets.js

Add local multi container setup to mirror production

Create a folder{watch} equivalent so we don't have to run a seperate process for development

Fix image profile retention not working in production and tell Simon

• maybe a caching issue and actually works?

Use tinify free api to compress some of the folder images

• maybe consider building this into the main product? it might pay for itself

Fix bug with first edit on forked template on template editor:

• 1. open template editor for fresh blog
• 2. change the text color
• 3. nothing happens (due to the inital fork) until you reload the page

Add asset library / file upload to template editor and share files across templates

• use this to help implement a favicon feature which minifies the favicon

Extend markdown parser to add a special class to a list of only images for targeting into an image grid or gallery?

• - ![Image](_image.jpg)... -> <ul class="image-list">... could then be used in CSS?

Extract multiple thumbnails per blog post (as many as there are images) for cooler image grid possibilities.

Add an open folder button to Local folder client

• maybe add a highlevel function which clients can expose to open a file (e.g. link to dropbox.com or google drive)?

Run image analysis on all images in post

• extract colors for searching
• use something to identify related images

Improve tools section

Fix hammering of github api for commit history for news page – cache locally to data directory

Improvements to questions section:

• Add 'blue dot' notifications icon to updates and questions based on last post
• Add colors to tags on questions section:
  • https://gauthierroussilhe.com/articles
  • https://designportland.org/
• Add a filter for topic index to show questions by: 'Newest', 'Active', 'Unanswered', 'Votes'

Move Dockerfile into app

Clean up scripts directory, lots of unused or broken scripts

---

Improve importer

Rewrite code which generates a fake 'macOS' folder to work cross platform:

• if the user is browsing from Windows create a windows folder
• if the user is browsing from linux create a linux folder

As much static file delivery through CDN as possible

Get folders out of build images

Start my own interview of customers 'behind the folder'?

• inspired by usesthis and p&b
• which tools they use
• how they organize their folder

Add auto template translator

Re-set up html minification using html-terser

Investigate whether openrestycache can be modified to pre-compress gzip responses

• can we add support for brotli?
• can we use https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html#gzip_static
• should we just be using varnish?

Get critique on full openresty config from LLM

Consider adding varnish in between openresty and nginx and get rid of openrestycache?

Improve developer documentation:

• Add interactive template renderer to developer docs
• Document page_size, sort_order, sort_by properties of templates

Drag and drop file and folder upload to dashboard

global site search of:

• files on site
• posts on site
• documentation
• questions
• templates

Investigate bug with link format using filename and follow up with yct

Overdue subscription payment form not working

• Possible bug introduced by recent stripe upgrade
• Migrate Stripe library to v7

Fix file sorting on folder on dashboard (clicking 'Date modified' e.g. does nothing)

Improvements to local folder experience

• Interpolate fonts based on an ID (a la web template editor)
• Interpolate syntax highlighting based on an ID

Fix bugs with template engine:

• Prevent double submission on template editor at insertion and front-end levels
• Make it possible to rename template directories and tell Darren and tell questioner
• Fix bug not exposing issue with invalid JSON in package.json for locally-edited template on dashboard and tell Jason
• Investigate bug with template views whose names include hashtags and follow up with Jay
• Fix bug with undeleteable template files prefixed with dot and tell Stuart
• Fix bug with template editor and ensure POST error messages are passed to client properly through ajax
  • to trigger, submit a >5mb file on web template editor
• Fix bug with chinese characters in template name and follow up with Alex
• Fix bug with date settings on the template editor for magazine template
• Fix date selector for template and tell Zach
• Fix bug with forking / editing new templates and tell Jarrod when local template folders better reflect template names

Use thumbhash to generate placeholders when lazy loading images and tell David and Marius

Add more efficient popular_tags method

• add new properties to underlying model

Surface recently edited posts on the sites tab of the dashboard

Fix issue with server restart:

• sudo chown ec2-user:ec2-user /var/instance-ssd/tmp
• sudo chown ec2-user:ec2-user /var/instance-ssd/logs/app.log
• mkfs -t xfs -f /dev/XXXXXXXX

Right now changes to documentation CSS do not make it into production:

• stale servers serve the old CSS
• purge bunny after deployment complete?

We should have a plan to switch CDN from Bunny to Cloudfront in an emergency

• Should be a simple DNS change

Can we support white-label cdn subdomain so there are no Blot CDN links?

Refactor helper/screenshot to use a persistently open browser

Change dashboard cookies: Set-Cookie: mycookie=value; Path=/; SameSite=Lax; Secure;

Write tests for animated gif thumbnail

Improvements to Google Drive client

• Fix bug with Mark's site which unneccessarily re-downloads some (all?) files on sync rather than using the correct hash and tell Mark
• new folder setup (e.g. waiting for user to share folder on drive) needs to survive a server restart (See init.js TODO)
• slowly resync all connected blogs on server restart
• It seems like sometimes the first sentence of Google Docs become titles unexpectedly – possibly a mis-identified title tag?
• make it possible to short-circuit the full folder traversal and immediately begin at a specific file/subfolder
• to improve sync speed, eventually set up temporary channels to watch google docs for changes – is this faster than polling driveactivity? or work out the fastest way to get fast updates for google docs. Is the move to listen on changes.watch and then setup an optional temporary files.watch channel to stream changes for the length of the channel (don't renew)?
• add tests with nock
• Test multiple sites sync in parallel
• Test multiple Google Drive accounts in parallel
• On disconnect, convert Google Docs in folder into Markdown and inform the user
• Test drafts
• Test local templates
• Resolve issue with Stephen's large blog
  • It also seems like server crashes when a massive google drive folder is synced, with a directory containing 6772 items
• How do we handle two files with same name and same parent? How does our fileId:path hash handle this? Make sure we delete the right file. Will file.title as opposed to file.name give us the deduped version?
  • Bug: move file with deduplicated name from sub folder to other folder
  • download should check for an existing file with same name but different fileId and dedupe before storing the path?
• Handle revoked credentials without tanking server
• Handle removed blog folder without error
• Handle renamed blog folder
• Can the google drive client handle a server restart during setup?
• database.folder should throw ENOENT and EEXIST errors for del, set, move etc.
• Error handling
  • Retry when we get google drive's This service is not available error
  • Handle storage quota errors
  • Handle 429 errors with exponential backoff retry
• Add a way for sync/build to accept a ctime as an option. We can't set this at the operating system level but google drive's API does return a createdTime value and we could use this as the true file's created date, rather than the date the file is made on the server
• write should use the best method for large file uploading
• Work out if we can test this using Mocks

---

Improve design of default error page

• https://kristianhjelle.com/

Investigate issues using git in production

Investigate why some blogs are accessible after deletion

Make templates/folders build less busy and then re-enable it in app/setup

• consider using git? use local client in smarter way?

Fix bug with pages not appearing in backlinks and tell questioner

Surface Blot client sync errors on dashboard, add email notifications for errors

• surface dropbox out of space error

Fix bug with magic text in posts which breaks backend and tell Victor

Fix bug with site export and contact Ryan

Fix bug with Git client and Obsidian then tell levelsi

1. create two files

source.txt "[[Target]]" target.txt

Add in one commit

2. rename target.txt to foo.txt

update in one commit

---

Use custom libvips so we can convert HEIC files

Expose exif data on Photo posts? https://github.com/lovell/sharp/issues/285 See if you can add comments to image metadata and tell Riley , also tell Mike

• are the source location and website of an image drag and dropped into finder stored in its exif data somewhere? this could be used to attribute the source of an image for personal image archives
• questioner (search exif)
• https://github.com/sambecker/exif-photo-blog

Think about migrating to s3 for blog data storage

• Purge thumbnails and cached images which are no longer in use
• Abstract methods which read, write and readdir the blog folder? static files folder?
• What about pointing the CDN to an s3 bucket then redirecting to upstream for 404? http { server { listen 80; # Main location block location / { # Proxy to the S3 bucket set $s3_bucket "bucket.s3.amazonaws.com"; proxy_pass https://$s3_bucket; # Pass the original Host header (required by S3) proxy_set_header Host $s3_bucket; # Handle S3 redirects properly proxy_redirect off; # If the response is 404, redirect to the fallback location error_page 404 = @fallback; } # Fallback location if the S3 bucket returns a 404 location @fallback { # Proxy to the fallback upstream proxy_pass http://fallback-upstream; # Optional: Set a custom Host header for the fallback upstream proxy_set_header Host fallback-upstream; } } }

Improvements to template engine

• Make it possible to query recent entries by tag and tell Peter
• Assign guid template IDs
• Assign unique path to template
• Assign template withs with full filename
• Make writeToFolder more efficient by not removing the entire folder each time (only remove orphaned files) and by no rewriting identical files
• Add search bar to template editor
• Make syntax highlight dark mode light mode compatible somehow?
• Work out why rebuilding template from folder clears cache even if the template is not changed
  • stop this
• Add new template lambdas
  • Whitespace trimming and tell Chad
  • Number formatting (e.g. add commas to int.) and tell Chad
• Make template slug seperate from the template ID, simply for human readability
• Tell bukit when it's possible to change the URL of the /tagged/x pages in the template engine
• Investigate performance of rich text editor in Safari on template editor and follow up with Rachel
• Make it possible to change URL of tagged pages and tell Jarrod
• Add blog.statistics and entry.statistics (total words, etc.) and tell Pratik
• Work out how to make live but saveable modifications for preview purposes on the template editor. I have already established a messaging channel between preview HTML pages and the template editor with the script injected in render/middleware.
• Add background patterns to template editor tools, tell Ish
• Add 'group by' function which would allow you to group posts by month/ year in the same way that {{#archives}} lets you and tell Jay
• Respond to Michael's thoughtful emails about the template system
• Add support for chronological template ordering and follow up with questions
• Add way to iterate over public and private pages to allow sitemap to work properly
  • tell Rodrigo
  • questioner
  • Tell Steve
  • Jack
• Allow the name/slug of a template to change and tell Marcus and Pratik and Brad when the name of a locally-edited template in your blog's folder updates
• Allow ordering of posts by file name and tell Jacopo
• Expose request query to views and tell Abhishek
• Add way to query list of posts using SQL? Something like jsql com/mail/u/0/#inbox/1588e3df7489b554)
  • Don't write my own DSL, use SQL Something like this in front matter? recent_posts: SELECT POSTS WHERE ...
  • Add support for entry lists of variable length and tell Logan
• Add support for the setting of custom headers (e.g. to enable CORS) and tell Roy
• Add support for listing files in arbritary folders (e.g. use a custom folder as a 'public' folder) and tell Nash and Marco
• Add way to list directory contents and tell Diego
• Move to data directory for templates
  • Serve the template directory as a static folder, allowing things like favicon.png and humans.txt to not become blog posts. Follow up with Jamie
• Re-do url redirector to use route-parser and properly support regular expressions
• Add support for paginated list of entries on other pages and Dan tell Kevin and Eli and Sven and Nikhil and Adam and Madino and Ryan
• Add dynamic querying of entry list and tell Naoki
• Fix bug with filtering paginated list of entries and tell Roy and Josh and Logan
• Add a popular entry list
• Add way to auto-refresh page during template development on preview subdomains. Tell Ryan
• Add way to list files at a path and tell Matt and Daisy
• General:
  • Add paginateable entries list
  • Add downloadable sites
  • Add static file uploader
  • Make templates folder based?
    • data/templates/blog_id/template_name
  • Add webfont uploader
  • Generate favicons of different sizes from uploaded image
• Fix bug with template tags in blog post and tell Shibel
• Re-enable partial templates in posts/pages
  • Tell Matt
• Add pagination to archives and tagged pages on default templates using a new tag
• Inject title metadata into h1 tag for all the text-based default templates and tell questioners
• Add 'download as zip' option to template editor
• Create starter kits for each template
  • e.g. a starter kit on the blog template for a writer, or a photographer, or a journalist, or an academic.
  • these could be downloaded as folders on the template page?
  • Get David's feedback when one exists for writers
• Create an index of customer-created templates and add Matt's theme
• Add 'Hidden: yes' metadata to replace 'Menu: no' and tell Sebastien
• Add method to merge updates into custom templates and tell Dave
• Redirect URLs with trailing slash when possible (i.e. not directory browsing) to fix possible duplicate content penalties and tell Marius
  • Add rel=canonical to all the default templates, or fix the issue that Blot serves the same thing to URLs with a trailing slash, and without.
  • If we do add rel=canonical, make sure to consider the situation where someone mounts a page to the index of their site, e.g. /page/1 and / will now be different.
• add support for dynamic partials to make collections possible, e.g. {{> /:tag.html}} ?
• Add neat interface to customize blog level variables?
• How do we transition from a global date display format to one that templates have control over?
• Consider making file-based template directory a git repo and using this to implement basic undo-redo functionality.
• Disable re-rendering for locals
• Add better customization options
• Local template editing does not clear cache
• Use Hogan for precompiling and rendering templates for performance?
• Parse tokens and params from routes and tell Jason
• Add way to allow all entry lists to access {{#months}} sorting options just like the archives page.
• Tell Marco when arbirary list sorting is ready
• Fix Frank's strange template bug
• Add support for fetching only some properties of an entry (ideally only those that are used). prevent some properties from being fetched in some lists (html, for example in all_entries). Would be a good candidate for hgetall or hget?
• Incorporate some of notebook's features in a template

For broken embedded images, attempt to resolve based on a shortest path approach, as in, traverse the folder and attempt to find an image with a matching name

• Tell Ellane

Make it possible to embed the HTML output of markdown files using Obsidians embedding syntax: ![[Filename]]

• Tell Ellane

---

Remove packages:

• node-fetch (use global fetch)
• twemoji (delete the plugin)

Make it possible to embed audio like so ![Audio]({{{path}}})

Experiement with injecting script into preview sites which can postMessage to embedding page, sending html2canvas result which can be stored locally for faster loading?

Make it possible to embed video like so ![Video](...)

Make it possible to embed PDF like so ![Video](...)

Merge comment and disqus into single comments plugin

Totally remove twemoji and emoticons plugin

Fix single line break handling in Word Documents and tell Paul

Make the 404 page more useful (see Netlify site)

Clamp the max length of blot usernames and custom domains

Expose markdown build errors on the dashboard to prevent weird non-update situations where it looks like the sync is stuck

Fix bug with bulk redirects page

Improvements to search engine:

• Make search use 'AND' instead of 'OR' for multiple terms
• Remove drafts and tell Paul
• Make search: true work and tell Paul

Make it possible to link between org mode files with: [[file:another-test.org][another test]] and tell questioners

---

Infrastructure plan:

• switch app deployment to ECS
• work on reducing the storage class of EBS
• create three processes:
  • blogs
  • dashboard
  • sync
• Use restic or similar to make incremental off-site backups of data directory:
  • restic -r b2:bucket-name:/path/to/repo forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 3 --prune

---

Fix issue with camelCase custom metadata and tell questioner

Investigate bug with org mode links and follow up with Elliot

Fix light mode colors and tell Gary

Fix Git client bugs

• Investigate a bug with git client credentials triggered when you have two sites using git on one account, and delete one of the sites
• Fix bug which occurs when you change your username while using the Git client (caused by the broken repo url) and tell Koen and Andre and Josh and Ryan and David
• Work out how to pipe custom error messages down
• Pipe custom message when you change a blog's handle (breaking repo url)
• Tell Koen about the broken repo URL fix

Add an option to use the date modified property of the file, rather than the date created property as the publish date. This would be especially useful for the Dropbox client, which nukes the original file creation date but preserves the client modified data. This should be opt in because it could lead to misleading behaviour where an old file is dragged into your folder and is published way deep in your archives. Tell Dirt

Fix bug so that script fix is able to remove tag items without posts

• how to reproduce this bug?
• it seems there is a phanton entry which no longer exists on a tag index
• can we iterate over all tag keys during fix to detect this?

Fix bug for 'edit this page' link in production

Fix bug with images zooming inside footnotes and tell Nathan

Fix bug with menu which occur when multiple pages are added at once Menu needs concurrency or atomic operations (so we can build posts in parallel)

Fix bug with .emf files in Word Document and follow up with Todd. Can we convert EMFs? Can we pass a flag to pandoc to let them deal with it? Is this just caused by Tiff files? I need a Windows machine to test this.

Come up with a workaround for Silas's Dropbox business issue

Fix issue with favicon and tell questioner

Can we use transformers.js for content similarity or auto captioning?

Fix bug with bandcamp embeds in Word Documents and tell Art

Fix navigation link bug with blog tempalte and tell Jason

Investigate this date metadata bug

Fix bug with this question

Fix issue with git client mirroring on GitHub by follow steps in email and follow up with n

Fix bug with the typeset settings do not actually affect anything (you can only disable/enable)

Fix bug with rename detector which caused old permalinks not to redirect to their latest version and tell Shibel

• To reproduce, create file 'foo.txt' with title and permalink 'foo'
• Then turn off server
• Then rename file to 'bar.txt', and edit title to 'bar' at the same time
• we expect /foo to redirect to /bar, but it doesn't

Fix bug with particular emoji and tell John

Fix case-sensitivity issue for relative (and absolute?) paths to files in folder in production and tell Marco

Fix bug with many twitter embeds and tell Nash

Fix bug with a once-scheduled post, based on the file's path or previous date metadata. The post remains scheduled even if the date is removed, and even if it is then rebuilt.

Fix bugs with RSS feeds validation. Perhaps extend absoluteURLs helper function? Tell John

Fix bug with date metadata on Vincent's blog and follow up. I suspect it's to do with the ordering of date tokens.

Fix bug with tokens inside blog posts and follow up with Karoly

Fix bug with potentially TeX-related escaping issues and tell Anthony and Roy

Fix issue with variable passed to dateFormat function and tell Leo

Fix bug with image zoomer on mobile devices and tell Alessandro

Fix bug with handling of queries and tell this person.

Fix post encoding bugs

• Fix Kevin's URL encoding bug. Each entry's url property is not encoded as a URI component on the server but it needs to be.
• Fix encoding bug with greek URLs then tell Rodericus know

Fix bugs with RSS feeds validation. Perhaps extend absoluteURLs helper function? Tell John

Fix bug with Rodrigo's Word Document files

Fix bug with image minification which produces larger file than source file and tell Jan

Date parser bugs

• Fix bug parsing ISO8601 date format and tell Thomas

---

Add textbundle support and tell questioner

• this will help me understand how best to accomplish multi-file posts

Document image zoom disable feature ?zoom=false and tell questioner

Add way to import directly

• Add/fix micro.blog import to dashboard and tell [Gabriel](https://mail.google.com/mail/u/0/#inbox/FMfcgzGwJRvfgrRhXwqDNDlxbpXmRlbd]
• And recurring import from elsewhere

Finish site and dashboard

• Fix live stream of folder sync status
• The site CSS persists in the browser cache
• Make sure folders are zipped when site builds, and delivered through CDN
• Make sure blot resyncs folders with their associated sites when pulled
  • Compute a hash of the folder contents and only resync or re-zip as needed
    • Think about mtimes, metadata as well as file contents
• The top right button 'your sites' should remember the last page of the dashboard you were on so you can switch from docs to dashboard quickly

Come up with OKRs for Blot

• Churn rate

Light mode/dark mode videos

Windows video + windows finder.css

Rewrite broken link tests

Improvements to templates:

• Fix bug with videos on Reference template and Tell Marco
• Fix post navigation on Portfolio template and tell Questioner
• Fix responsive thumbnail component on photo template (it loads large thumbnail every time)
• Fix 'back' button in photo template and tell Bibi
• Fix side navigation on Blog template and tell Sava
• Fix bug with date on magazine template
• Add layout fix to blog template and tell Lewis Fix bug with header position on the blog template
• Library
  • mobile
  • make links open immediately
  • fix archives page
  • rename from reference
  • move tags into top menu
  • add tag filtering (nested tags)
• Portfolio
  • mobile
  • spacing issue on firefox
  • add full control for navigation font and logo font
  • add infinite scroll
  • add filler nodes for google images grid so you don't get inconsistent sizes
  • add control over navigation width
  • Improve typography of text posts
  • improve navigation between posts
  • fix boolean options missing (e.g. hide titles)
• Magazine
  • mobile
  • rewrite in plain HTML
• Image
  • mobile
  • rename from photo
• Make 'close' button on entry permalink page of reference template work like the photo template and tell Marco
• Add scrubbing pagination to images template

Add pause subscription

• Enable hibernation of blogs

Add disclaimer to pricing page for old subscribers

Add Generative tree for account page

Add support for INI files and tell Frank

Add {{#posts-without-:tag}} and tell steve

Improvements to website:

• Fix broken link checker
• Create material in docs with title 'Turn X into website' where X is:
  • Dropbox, Google Drive, etc.
  • Word Document, Google Doc, etc.
• Document text editors https://coteditor.com/ https://zettelkasten.de/the-archive/ https://tot.rocks/ https://desairem.com/wordpress/ufocus/
• Add guide to useful text editors like this one recommended by Andrew
  • do this as a marketing strategy, e.g. target 'blog with microsoft word', 'blog with google docs', 'blog with ia writer' etc... contact the people who make text editors and ask them to link to us?
• Personalize the documentation for logged in customers
• Add contributor / language and repo information to source code page using GitHub API.
• Add upload to dashboard
  • Drag and drop
• Redo demo video

---

Consider switching domain to blot.site

• add support for different user subdomain in dev?
• set up mail
• switch from mailgun to SES over new domain

Add aws cost usage and forecast to daily update email

• https://docs.aws.amazon.com/cli/latest/reference/ce/get-cost-and-usage.html

Make it possible to ask a question without logging in

Can we support google docs even from Dropbox? if the doc is public?

Investigate why changes made to app/scheduler/daily/revenue.js increased the number of active blogs in my email

Ensure that overdue PayPal subscription webhooks map to notification emails

Final polish on stats dashboard

• Add monitor the root disk usage of both servers to stats
• Tie in daily update stats to stats dashboard

Purge old docs CSS and JS from CDN storage cache

Make sure main server can hard restart without issue

Data use minimization:

• Write automated script to archive overdue blogs from disk and db
• Write script to remove unused and unaccessed static files from within blog folders
  • we'll also need to purge the cache of the transformer module for the file

Set up fail2ban on nginx logs

When a post is deleted, we should remove everything except the created time?

Create a queue for purging cache, group requests together to prevent stampede

Prevent overzealous purging of cache for syncs in which the folder does not change

Add WebDav support

Improve paypal integration:

• Get create new blog working
  • Add way to modify subscription quantity for paypal users
• Create new router for paypal subscription settings
• Test subscription with real paypal account
  • Test new blog creation
  • Test new blog deletion
  • Test switch to annual billing
  • Test account deletion
• Add subscription pause
• Make it possible to add paypal as a payment method for existing stripe users
• Verify signature integrity of paypal webhooks
• Use PayPal buttons api to modify subscription properties
• Get account page working
  • Add way to cancel subscription from dashboard
  • Add way to delete account from dashboard
• Update scheduleEmailNotification for paypal customers

Can I use proxy_store instead of openrestycache?

Make {{active}} strip the domain from the link before checking? Or automatically strip the domain from internal links on Blot's dashboard?

Can we add a 'folder rules' section to create ignore globs, multi post globs, etc.

When a folder has lots of elements, trim the list of files on the dashboard, at least on the index page

Store path -> md5 hash on sync

• Use to auto version links to files in your folder so they can be cached

Investigate videos instead of gifs for thumbnails

Eventually configure cloud watch reboot sequence then test it

---

Investigate if we can rewrite git client so we don't need to store double the data - right now we have 48GB duped.

Fix auto titles of 'screenshot at 12:22pm' and make sure the date parser can handle them

Fix title generator so h2 titles within the file dont become the title of the post – this is helpful for obsidian

Get to bottom of issues with questions/tests/list and re-enable disable test suite

The password check on the delete blog page does not print the error message if the password is incorrect? Or just for admin

---

Improve image posts

• Add support for large image tiling and zooming with photoswipe

---

Server improvements:

• Copy the cron tab into the new server setup script
• Add more stringent checks to scripts which mount instance storage and external volumes
• Install fail2ban on proxy logs
  • we don't need to worry about banning CDN ips because those requests go straight to the origin server
  • this should reduce origin load
• Get blot proxy communicating with node process via unix socket
• Understand why the server gets so slow when we switch from SSD ebs to HDD ebs
• it would be ideal to use HDD for bulk of blog storage, and take advantage of the local SSD for high performance operations
  • Investigate use of aws s3 sync for backing up ebs volume rather than snapshots
• Test reboot for all servers

Add support for proxying particular analytics files and tell questioner

One bad latex equation should not tank the build process for the whole post

Follow up with Tim about activitypub

Work out why we get a 'Zombie process' notification email every morning at 9:05am – now 8:05am it seems the process hangs

Wikilinks should have a class property of 'wikilink' rather than a title attribute and tell questioner

---

Wikilinks should have a class property of 'wikilink' rather than a title attribute and tell questioner

Preserve file name for SEO purposes when uploading to image cache CDN and tell Jean

Add a property to entries which lets you list all the images in a post so Jean can add them to his sitemap

Add lazy loading option to image settings and tell questioner

---

Improve CI testing

• Add spell checking test
• Write browser based integration tests for core functionality
• how can we test the 'reload-server' script?
• how can we test the 'scheduler' functions? we ran into error where it was running twice

Methods to improve server performance

• consider expanding the replication of the perma_cache across more regions
• consider switching to more expensive CDN type (i.e. not volume)
• switch to more expensive EBS for disk of server

---

Add support for paying with crypto, BTC ETH

• Consider https://github.com/btcpayserver https://github.com/btcpayserver/node-btcpay
• Tell Chaz
• Tell Anita
• Tell Arc
• Tell Dmitriy

Improve Are.na importer so it handles descriptions + also other file types (e.g. PDF, text) and tell Marco

Improvements to server

• move development cdn onto its own host to better simular cross-domain issues
• test maintenance and redirector on dashboard
• restrict post to /dashboard route so we can tune nginx logic?
• tune performance of express-mustache in production and development
• add optional HTML/JS/CSS minification to express-disk-cache
• Test overdue subscription re-start
• Create blog > Dropbox > Create blog > Dropbox failed to set up properly
• Prevent GET request to log in page from setting connect.sid cookie
  • or otherwise work out how to try and cache everything at nginx
  • log in route should clear 'logged_in_to_blot' cookie if not logged in
• Use Lory to add a slider to the homepage

Improvements to sync/reset scripts:

• Fix should produce a report which I get email
• Resync should produce a report which I get via email
• dropbox reset TO BLOT should handle removed folder
• dropbox reset FROM BLOT should handle removed folder
• dropbox reset TO BLOT should handle revoked token
• dropbox reset FROM BLOT should handle revoked token
• make dropbox reset FROM abortable
• make dropbox reset TO abortable

Improvements to Dropbox client:

• Determine if we can hack our way to determining the file upload date on Dropbox using their API?
• Parallelize build/download a little more, so posts appear when many files are dropped into folder?
• Efficiently handle Dropbox file renames with internal hash of fileIDs
• surface errors during setup on dashboard
  • surface sync issues with Dropbox client on the homepage of dashboard
  • How do we make errors sticky? To make sure users click through to clients page to re-connect to Dropbox, etc...
• run reset-to-blot when sync fails / errors
• ensure that when a process dies, any blogs syncing will be reset-to-blot
• ensure that when a process dies, any blogs setting up db client will be re-attempted
• reconnecting with the same account and same permissions should not create a new folder for full_access=true
• make full_folder use the old app folder directory – is there a way to disconnect it from the Blot app? otherwise add a big warning to prevent confusion about where the new folder is when transitioning permissions
• handle the download of large files gracefully too
• consider adding back in download/upload stream, but add a check to see if the download is stuck then fall back to other method?
• add a check for folder storage before doing mass upload
• handle folder full error appropriately
• handle a sync lock failure on second blog when migrating an app folder
• check content hash for upload possible

--

Add support for Indian payments and follow up with Indian customers

• Pradeep wants to join

Add support for Obsidian media embedding

• ![[_image.png]]
• https://help.obsidian.md/Linking+notes+and+files/Embedding+files
• Tell Kevin
• Look into Wikipedia's blended wikilinks
• Update response to question about adding class="wikilink" replacing it with title="wikilink"
• Tell John
• Tell Robert
• Tell Levo
• Tell kevin
• [ ]Follow up with Marco about: ![La Escocesa 2006-2010](files/_laescocesaold.png) ![[files/_laescocesaold.png]]
• Tell Matthew

---

Look into Pandoc's RAW HTML extension for resolving issues mixing HTML into markdown files:

• markdown_in_html_blocks
• raw_html
• native_divs

Move Wikilinks to Pandoc

• Update version of pandoc
• Support Obsidian's syntax for embedding files and images, and tell John and Robert and Levo
• Follow up with Marco about: ![La Escocesa 2006-2010](files/_laescocesaold.png) ![[files/_laescocesaold.png]]

Add explanation for which files are causing which menu links

Add export button to overdue subscription page

Add way to restore a blog from export

Add way to escape commas in tags and tell questioner

---

Investigate ignored files situation

• does rebuild update ignored files list too?
• handle new pathNormalizer behaviour
• 'ignoredFilesKey'
• why are ignored files not added to ignored list?
• add 'ignored ghosts' check to sync/fix

Improvements to importer

• Can we offload an import into a resource constrained child process?
• Can we queue up imports and process them one at a time?
• Add interface to move import into folder directly
• Add scheduled imports on regular basis into folder directly
• Add nice import step under 'Services' and as part of sign up
• Fix bug with script to import wordpress posts and follow up with Simon
• Improve Ghost importer and tell Vin
• Create importer script for John's posts from flickr
• Create importer script for tumblr text files and tell Sawyer
• Create importer script for Weebly and tell David
• Map excerpt metadata to summary for Jekyll converter importer
• Look into how Hexo has written imports for blogger, joomla, rss, wordpress.
• Add an importer for Twitter and sell Sawyer

Add 'confirm email address' flow to prevent typos in emails

Add simple form endpoint, could be used for contact forms

Add theme with photoswipe and use sharp's tile generation feature along with

Add Ignore: yes metadata to trigger the ignored file behaviour

Add hot reload for preview subdomains

Add way to set the slug-part of a URL for a post and tell George and follow up for post in Questions and tell Jarrod

---

Can Blot blogs be federated? Respond to Nigel once I do some research on this

Rebuild folder sites and tell Michael that the broken image link is fixed

Follow conventional delimiters for KateX

---

Support larger files on DB client and tell Marco

Add support for new formats:

• jupyter notebooks (via pandoc) and tell questioner

---

Explain that the git client uses the master branch and repos from github will use main in the docs and on the dashboard then tell N

Send out cancellation confirmation emails with option to give feedback

• auto-refund people who immediately close their account after signing up

Get to the bottom of the slow cloudflare issue and follow up with Wen

Git client improvements:

• Fix issue which occurs when you change your username with the git client
• Fix issue with switching from Dropbox client to Git client, in production which affected Nat

--

Add tag to iterate over all images in a folder and tell Laurel

Fix bug with Work out why Pandoc cannot handle 'linked' images (instead of conventionally embedded images). For instructiosn on reproducing bug, see SG's email and follow up when resolved.

Fix bug with tags that contain slashes and tell Jack

---

Improve backlink's tolerance

• Add support for full qualified URLs in the backlinks list for each entry. For example, if your blog is example.com, the following link will not appear as an internal link: Foo because build/prepare/internalLinks is not aware of the blog domain/handle
  • Tell Eli
• Add support for internal links without a leading slash
  • Tell Laurel

Add support for 2FA (Two factor authentication) then tell Luke, Max, Chaz, Prathamesh and Tom.

Add support for SSO then tell Prathamesh

Make retreiving files from server case-insentive

• i.e. add resolve case insensitive path to file to blog/asset.js

Prevent full disk from affecting access to people's blogs.

• switch into a read-only mode?
  • the specific issue that takes down the server seems to be the redis error (cannot persist on disk, out of space)

---

Improvements to dashboard:

• Smooth over the custom domain adding process
  • cache domain validity and display checkmark on dashboard
  • validate the domain is pointing to Blot before allowing resty-auto-ssl to request a cert. we seem to get into a DNS mismatch where a client thinks a domain points to Blot, and triggers a cert request through resty-auto-ssl but whatever DNS server Blot's server points to has the stale record (pointing elsewhere)
  • Disable redirect to HTTPS for blogs by default (to ensure old browsers without SNI work) but ensure all prominent links to the blog on dashboard use HTTPS
• Improve clients
  • cache sync validity and display checkmark on dashboard
  • surface errors for clients on dashboard index
  • add a change button next to the dropbox folder selected on the dropbox client where possible
  • add blocking process to dropbox connection when writing folder contents
  • ensure the app permission access_token is destroyed when switching to full-folder and vice versa
  • add a folder chooser during full-folder setup
  • ensure you can move the folder in full-folder permission mode

Move citations out of {{{html}}} and tell questioner and Mikka

Add OneDrive client

• Tell Nenad
• Tell John
• Tell Shantesh
• Tell Nathan
• Tell ekskog
• Tell John
• Tell Simon
• Tell Andrew
• Tell Naveen
• Tell Emad
• Tell John
• If OneDrive for Business is supported tell Josh

---

Re-order blog engine routes such that pages/entries are evauluated first, making it possible to set up custom redirects and tell Marco

---

Write new converters to turn files into posts

• Add AVIF image post support and tell Simon
• Add support for Jupyter Notebook .ipynb using Pandoc and tell Arnab
• Add support for .doc using libreoffice and tell Kyle
• Video posts (.mp4) - add way to opt-in to these
• Add support for .tiff images
• OPML files and tell John
• Audio posts (.mp3 for now?) and tell Kevin - add way to opt-in

Write new clients to sync blog folders

• rsync
  • Tell Harry
• SFTP
  • Tell Coraline
  • Tell Calvin
  • Tell Fabrice
• NextCloud
  • Tell Vincent
  • Tell Coraline
  • Tell Andrew
  • Tell Mike
• Box
  • Tell Eric
• Syncthing
  • Tell Jason
• Github
  • Tell John
• Jumpshare
  • Tell Ömer
• pCloud
  • Tell Omaid
  • Tell Anders
  • Tell Tim
  • Tell JMN
• Dropmark
  • Tell Rado
• Proton Drive
  • Tell Ted
  • Tell Francois

---

Improvements to backup system

• Add hourly local redis rdb dumps to /cache
• Test maintenance mode
• Add script to roll back into hourly local db dumps and remote backups

Apply helper/imageminify to images in posts with opt-in

• Run these changes past John, get opinion on optimized images
• Talk to Marius about his opinions on the compression settings

Begin collecting proper measurements for page-rendering speed on the server

Add raw-link -> open graph cards/embed function, which would fetch the title, description and thumbnail for a link using tags. Tell Robert

Improvements to teaser generator

• Improve estimate of teaser length
• Remove trailing non-text elements (like

  ---

  )
• Add {{small}} and {{long}} versions of teaser?
• Fix bug with makeTeaser for ~/Dropbox/bug.html which locks main process painfully
• Internal links (e.g. #section) in {{{teaser}}} should be resolved against the entry's permalink.
• Add hasBreakpoint property to entry to allow greater control over display of {{{teaser}}} and tell Pratik
• Fix issue with teaser generator that locks up a build process, sometimes for 10+ seconds.
• Make it possible for users to select length of teasers and Leo

Performance improvements

• Consider switching redis client, perhaps to ioredis?
• Use a docker image to speed up test suite: https://github.com/davidmerfield/Blot/pull/463
• Faster server, with SSL termination across CDN
• set up 'cdn.blot.im' as cloudfront distribution endpoint?
• Streamline and cache loading of data on dashboard
• Ensure node.js server has one process per CPU
• Detect CPU intensive functions in node.js process
• Use the --trace-sync-io command-line flag to determine if Blot is using any sync methods inappropriately
• Can we keep the pandoc child_process running and pipe stuff in and out for speed? Measure before after to ensure this helps
• Add request level timeouts to prevent server swamping
  • ensure to remove the timeout from the SSE streaming routes (drafts, sync status on dashboard)
  • read this guide: http://expressjs.com/en/resources/middleware/timeout.html
  • it seems we also need to cancel the resources involved in responding to the request (i.e. disk io, db calls) for this to be really useful
  • Add a timeout of 15 seconds for building posts.

Server changes

• Write useful benchmark tool for Blot
• Work out how to detect stray build processes. There were a bunch of zombies consuming memory on Blot's server. Work out how to kill them safely
• Experiment with different EBS types, perhaps try to use ephemeral disk more since lower latency?
• Take full advantage of attached SSD
  • use rsync/inotify to synchronize 'source of truth' EBS disk with the local SSD cache then modify NGINX to check the synced folder on the /cache disk
  • on boot we would gently copy across all /blogs folders and the /static directory in background task

Add line numbers to highlight.js hljs code tags

Convert Vitepress to Blot and tell Nicolas

Expose log files on blog's dashboard and tell Roman, Sönke and questioner and John

Add end-to-end tests for brochure site to test sign up flow (use Playwright? Phantom? Chrome headless)

Improve wikilinks lookup algorithm:

• Add support for 'shortest path where possible' on Obsidian
  • search current directory for filename
  • search subdirectories for filename
  • search parent directories for filename
• Add support for 'absolute path' on Obsidian when the vault is a sub directory of your blog folder
  • How best to accomplish this?

Support Obsidian-style media embedding and tell questioners

Improvements to simple local server (npm run local):

• Consider using redis-mock and mockery
• Persist data somehow? import/export from local folder .blot?
• Fix 'Visit' link on dashboard
• Can we expand this to a simple self-hosting process ('npx blot-server')?

---

Fetch dimensions of images even if optimize and cache is disable, so that image zoom can work. Tell Joshua

Add action to lint code with jshint and eslint

• any other static code analysis tools?
• separate code to check for broken links, run in parallel

Improvements to metadata extractor:

• Improve date parser to use time from file creation date if it's within the same day as the date specified in metadata and follow up with Nicolas
• Fix summary generator such that a subtitle is used for summary text (i.e. an h2 directly following an h1) We still want to prevent the scenario in which a later h2 becomes the summary, though. Notify Jessica and rebuild Jessica's site
• Add support for entry property with escaped double quotes, tell Jay

Reduce image compression settings and tell Joshua

Improve questions section of site, in approximate order of importance:

• Add way to mass-rename tags
• Add display aliases for tags e.g. how -> How to use Blot, etc.
• Store version history of the question and answer edits
• give the 'ask' and 'answer' textarea inputs the full functionality of the GitHub issue/PR comment inputs (i.e. code snippets / preview / copy/paste screenshots from clipboard)
• email notifications (There's an existing system in place for sending emails in app/helper/email)
• upvoting for topics and upvote-based ranking on the topic index
• show vote count and answer count next to each question on index
• 'run code snippet' feature / similar to trinkets.io

---

Add way to nest tags / hierarchy for tags and tell Uzoamaka

Adjust the chargeForRemaining function in create-blog such that the balance is added or deducted from the proration charge.

Rewrite transformer (in app/helper/transformer)

• Speak HTTP properly, respect backoffs, use/send eTags
• Share information across blogs?
• Improve transformer so it checks a file's MTIME against a stored value before rehashing the file?
• Adjust interface to accept a number of directories. Make it 'unaware' of blog IDs?
• Add method which accepts a path or URL and flushes what is stored for it, update the rebuild entry script
• Cache errors. When a post contains multiple identical links to images that timeout/404, the thumbnail generator should only make 1 attempt to fetch them. Right now we have a post that is filled with identical broken image links (in this case to a spacer.gif) can suck up the build process for hours.

Add a 'page link format' option to the dashboard – the link format now only applies to entries. Tell Mike

Prevent long-overdue customers from accessing the service, then tell Matvey

Generate proper favicons from uploaded avatars (to prevent streched favicons when non-square avatars are uploaded)

Experiment with GitHub actions setup

• Look into potentially useful actions for code quality / linting / automtatic dependency updates
  • super-linter?

Improvements to server infrastructure

• Work out way to recover from expired cert in the cache. To reproduce, consider a member who signed up, connected example.com, then moved the domain to another service. The attempt to renew the cert for example.com will fail. When the customer re-registers with Blot, for some reason open-resty-auto-ssl will serve an old certificate. I followed these steps to reset but we should automate this
• Create DMARC record for Blot https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail https://docs.aws.amazon.com/ses/latest/DeveloperGuide/dmarc.html
• NGINX doesn't appear to be following the symlinks created by node.js to serve the contents of each blog's folder and the contents of each blog's static directory. Is this a permissions issue? Is now finally the time to set up users with appropriate permissions for blot, nginx, build etc...

Improve the automatic title generated for screenshots with default MacOs filename from 'at 14:40pm'

Add support for posts generated from Excel files, would generate a table. Use this library?

Work out how to cache 404s at the NGINX level

• I tried extending express-disk-cache to store images inside /https/blot.im/missing/foo-bar.html but I can't set a 404 status code for the specific tryfiles directive which could match that file.
• The application server ends up spending a lot of time serving 404s

Re-send receipt when the customer changes their email after initial payment, pre-account creation.

• Once stripe allows programmatic receipt delivery

Improvements to entry builder

• Fix bug with slashes in tags and tell Dave
• Generate a list of titles which you could use to generate a table of contents and tell Zach
• Use time draft was moved into blog folder for entry created/publish date, rather than the time the draft file itself was created. Tell Kerim and Shibel
• Parse Word Document subtitles as h2 and tell Matt
• Add way to filter entries by post type (e.g. photo) and tell Justin.
• If entries share publish date, sort them by blot creation date
• Fix bug with blockquotes and tell Kerim. Multiple ">" symbols on the blank paragraphs between other quoted text are not formatting properly.
• Fix bug with indented code in pre tags and tell Michael

Improvements to dashboard

• Fix bug with file list on dashboard in Safari and tell Ryan
• Fix missing error messages on overdue subscription payment form
• Use <select> optgroup on timezone select dropdown to subdivide zones by current time, and steal the presentation format used by basecamp.
• Add a broken link checker
• Make dashboard requests idempotent to prevent multiple form resubmits, multiple new blogs created accidentally? More information about idempotent requests
• Indicate that some posts' permalink will not change (if metadata) on the dashboard when setting the blog's permalink format. Tell Chet.
• Dumb quotes are still smartified, even without typeset enabled, possibly by pandoc?
• Undo option for form changes
• Add transactional emails notifying customer of changes to their email or password.
• When a customer pastes the whole tracking script into the preferences page on the dashboard, strip out everything except the ID which we need... perhaps create a page explaining what is happening before doing this.
• Handle noscript on dashboard / signup gracefully (remove all dashboard js?)
• Allow customers who are restarting an account with multiple blogs to disable some before restarting.

---

Add password protection for blogs then tell Ed, this person, this person, and Will, and Modomiro and Julian and Raph and Jeremy and Tara and Phil and Florian and Roy's friend Tim and Kaleb and Lotus and Khaled and Noel and Franz and Justin and Kyle and Devan and John and Anita and Emerald and Ben and Giuseppe and Annie and Jan

• Should we just go all the way and support htaccess? Add URL rewrite feature, e.g. rewrite /randomColor to /public/randomColor?

---

Improve the markdown converter:

• Add plugin option to enable hard line breaks in Markdown parser? Tell Wil
• Add plugin to autogenerate anchor heading links and tell Peter
• Fix bug with nested Markdown and tell Martin:
• Extend the Markdown converted to add support for Ulysses highlighting and iAWriter highlighting syntax, using the example files provided by Marco. Tell Marco
• Fix bug and tell Vanessa which injects <p> tags

Finish script to identify long-expired customers with blogs to safely delete and free up space.

• Sort by date expired
• Write script to safely delete blogs
  • wipe blog folder
  • wipe static directory
  • wipe all keys

Ignore files inside 'assets' inside .textbundle directories to improve support for textbundles and tell Thomas

Rebuild all the thumbnails and cached images for every blog

---

Add support for paths to static files with a trailing slash then tell Sean

Add copy button to code snippets in <pre> tags and tell Atif

Add support for Ulysses .ulz file format and tell Nash

Add support for Ulysses image syntax and tell Konstantin

Finish template translation offers

• Anders Norén
• Radek Kozieł
• Port Jane hugo theme to Blot and tell Jack and Josh on micro.blog
• Ghostwriter and tell Simon
• Work on Tim's template
• Translate this theme to Blot and tell Naveen
• Translate Ana's designs to Blot
• Translate 2019 to Blot and tell Gabriel
• Translate Paul's template into Blot
• Add a bootswatch template and tell Michael
• Port the jekyll template Neocactus to Blot and document the process then tell Robert

Add way to filter archives page by year or month and tell Chad

Add a cool bookmarklet which drafts a post a la this demo

Follow up with Shibel about how to ignore arbritrary paths and keep clean URLS (no underscores). Sitewide .blotignore file?

Adjust length of slug to use as few words as possible until collision and tell Chet

---

Improve scheduler

• Adds a command to restore database from backup.

Remove dependence on AWS

• Adds offsite (i.e. non AWS backups) backups
• Write out plan to move off AWS
  • DNS
  • EC2
  • S3 for backups
  • Cloudfront for CDN
  • backup blogs folder + db somewhere offsite at least once per week

Metadata parser

• Make custom metadata more strict, follow up with Chris](https://mail.google.com/mail/u/0/#inbox/FMfcgxwCgpdRvrccbzfckkZRtPXHzqjs)
• Add support for Pelican-style metadata in HTML posts using
• Removing Date metadata from a post should reset its date to the file's creation date. Currently nothing changes.
• Fall back to file name before first sentence when generating title metadata and tell Rob
• Add support for setting the index of an item in the menu to metadata and tell Silvio

Implement hidden link section on Menu page under profile settings. Default links not on the menu would appear there, making it easy to restore them in future. Tell Josh

Add new variable for last-build-time for a blog, and use this for in the sites' RSS feeds. Tell Shawn

Redirect old draft URL to live published post and tell Shibel

Improve 404 log page on Dashboard

• Add button to clear 404 log and tell Jamie
• Add frequency indicator for each route 404 log and tell Jamie
• Add filtering by date to 404 log and tell this person

Add support for nesting multiple layout tags e.g. "{<>} {||} Y" and tell Marius

Look into accessibility issues with scrolling and code blocks and tell Eli if I make any changes

Improvements to Git client

• Make it possible to connect github repo via their API and tell reg
• Add support for LFS and tell Jamie
• Remove git folder when switching to Dropbox client and tell Jamie
• Backdate existing commits (perhaps set file ctime using info from git log?) This would sidestep the frustration experienced by Gavin when he imported an existing repository.
• Fix bugs with git removed / renamed file unit test on Travis
• Extend sync lock during long syncs
• Write test to simulate process dying and restarting mid-pull. Does the client recover? What happens if a pull fails midway? Does git recover?
• Write tests for a series of random client operations (pull, add, push, move, reset, delete) that can be seeded and reliably re-run in the case of failure.
• Move to neat promise chain api when 2.0 comes out

• Test that client must keep sending new entries and deleted entries when it recieved an error from entry.build otherwise you get ghost posts...

• send nice error message with instructions for 413 request too large: "git reset HEAD^" etc...
• Thoroughly document:
• Tidy up tests and reduce size and runtime
• This client should acquire a lock on the blog's folder before doing stuff to it more generally, especially during initialization etc...
• what happens if customer changes their handle? will pushing break?
  • I want them to be able to keep using their old one until someone else claims it, then I want 401 unauthorized, please reauthenticate with new username... but the remote path will be wrong?
• Rate limit authentication route?

Dropbox client

• Handle file renames using Dropbox IDs to avoid the 'new entry' bug when you rename a file
• Build posts incrementally while big syncs are taking place
• Revert lowercase for all files when disconnecting the client
• add support for Dropbox Paper files using paperDocsDownload and tell Emad
• Add integration test with a dropbox Webhook call. I want to not push some code which would prevent syncing customer folders.
• Does client.remove support the removal of folders? If not, make it so and add a test
• Add a step to authentication flow which says: 'there is already a folder called 'Your blog title' in your Dropbox, would you like to use that?' then overwrite the files inside. This prevents the proliferation of confusing 'Your blog title (1)' when you re-authenticate.
• Add a progress bar to the dashboard when writing existing files to Dropbox. It takes a while and confuses people.
• Adding an additional blog to existing app folder under constant webhooks failed. New blog had correct folder but old blog still used root. I guess we need to acquire a lock on ALL blogs interacting with that dropbox account.
• Extend sync lock during long syncs
• Write test to simulate process dying and restarting mid-pull. Does the client recover?
• Write tests for a series of random dropbox client operations (pull, add, push, move, reset, delete) that can be seeded and reliably re-run in the case of failure.
• Look into getAccessTokenFromCode on DBSDK repo to avoid the mess in get_account.js https://github.com/dropbox/dropbox-sdk-js/issues/64
• Gracefuly retry wrapper around Dropbox
• Can we write to a folder_id instead of a path? It would be nice to be able to move the blog's folder even when writing the initial files to it, for example...
• This should be simple once we copy the exisiting state of the blog folder across. But right now it doesn't really do much. It resets the local server folder. Then resyncs.
• Update to latest version of Dropbox client.
• Make the folder in Dropbox rename automatically when the customer changes the username for one of their multiple blogs?

Add media-query to prevent typeset hanging punctuation on small screens (it looks weird) and tell Shibel

Make Redis performance improvements

• Research Redis clustering
• Prune redis keys and begin storing more on disk
• redis analysis tool
• Move to using unix socket with redis client per documentation for better throughput. Read this issue first, I might need to make some user group permissions tweaks to ensure things are secure.

Restructure application folder

• make serving static files from NGINX easier?
  • ln /data/blogs/{blogID}/folder /data/cache/{hostname}/folder/...
  • ln /data/blogs/{blogID}/static /data/cache/{hostname}/static/...
• consolidating the data stored about each blog
• Specfic ideas:
  • Move cache directory from /cache to /data/cache
  • Move blog's folder contents from /blogs/{blogID} to data/blogs/{blogID}/folder
  • Move static files directory from /static/{blogID} to /data/blogs/{blogID}/static
  • Move db directory from /db to /data/db
  • Remove /tmp directory and instead use os.tmpDir

Switch to streaming JSON parser. There are npm modules that offer asynchronous JSON APIs. See for example:

• JSONStream, which has stream APIs.
• Big-Friendly JSON, which has stream APIs as well as asynchronous versions of the standard JSON APIs using the partitioning-on-the-Event-Loop paradigm outlined below.

Add notification emails for monit events

• Include secure one-time link to reboot the AWS instance?

Investigate support for xattr to read image metadata on Blot? Or source for dragged images?

Set NODE_ENV environment variable to production

Add support for HTTP2 and tell Ryan and questioner

Could we have multiple syncing method attached to one blog?

Generate list of external API dependencies and follow changes for each

Subscription refactor

• Different payment methods per blog
• Seperate blog subscriptions behind the scenes to avoid the crazy calculations. Think of them like domains, you wouldn't do what Blot is doing for those...

Create script to export Blot sites to common formats like Wordpress and let Roberto know

Move to Linode?

• Preserve AWS elastic IP and proxy to Linode
• Think about eventual multiple locations around the world from day 1
• Work out the pricing model carefully
• Use blockstorage this for storing all customers' folders
• Use backups
• Use nodebalancers

Increase price to $5 per month for new customers. This will not affect existing customers in any way.

---

Create an API for Blot

• Add webhooks and tell Aurynn and Shawn

Allow logged in customers to see public homepage. Perhaps use dashboard.blot.im for dashboard?

• Add a slack.com style dropdown on the public homepage for logged in customers which takes them to the dashboard for their sites.

When the customer's blog does not have a client, client.write and client.remove need to remove the files from the source folder. Create some kind of local client which we fall back and use when none is configured?

Work out how to host podcasts on Blot and tell Adrian

Client.write does not propogate sync errors... to demonstrate this, revoke dropbox token then try to enable local editing.

Re-enable image minifier and fix the bugs it creates

Prune package.json

• To minify HTML
• To minify CSS (alternative)
• To minify JavaScript (alternative)

Reconsider default slug length or add new short slug property and tell Josh

Write self-hosting documentation then tell Jake and Berndl and Joel and Mohammed and BP and Rado and Nuno and Omaid and Brian and Jarod and Francesco and Pranay and Aryan and Todd and Anita and Kaushik and Michael and Frank and Zip

• Tell Nick when it's possible to deploy a self-hosted Blot on Vercel and Zia

Add option to sort posts by filename alphabetically and tell Steve

Hidden posts:

• Fix _ entries to true public files, tell john & other folks who depend on the menu: yes, page: no side effect
• Warn customers that underscore will not work for hidden pages
• Tell Rodrigo don't write previews for _ posts

Embed feature

• Add support for native PDF embeds in posts, just put a link to a PDF or a path, e.g. /_assets/file.pdf --> would become a nice embed.
• Add support for native MP3 embeds in posts, just put a link to an MP3 or a path, e.g. /_assets/file.mp3 --> would become a nice embed then tell Luke and Nash and Andy and Charlie and Konstantin
• Add soundcloud embeds then tell Jonas, Nashp, and Anders and Richard
• Add are.na embeds and tell Marco

---

Improvements to thumbnail generator

• Prioritize faces in crop?
• Try to crop out ruined borders. E.g. when making a square crop of a rectangular image with a border, crop into the image as well to remove the border-on-three-sides effect
• Extract color palette too for the image
• Offer imgix style API for embedding thumbnails

Improvements to reverse proxy:

• Ensure I am doing request coalescing on NGINX and lower proxy timeout limit
• Read the 5 hour CDN post and discussion
• Ensure nginx is handling all static file delivery
• NGINX Install nginx fail2ban & http2 & reasonable rate-limits
• Add support for IPv6 and tell Raul

Add support for comments via webmentions like Disqus and tell Amit and Luke

Migrate code from require('mime') to require('mime-types') since it doesn't use a global default type. Remove the mime node module.

Plugins

• Fix image caption plugin to be semantic using

• Ensure Youtube plugin still functions if apis fails

Think about responsive images and optimizing at request time and tell Marius and Jan

Reply to Jamie about gitwatch and other newsletter suggestions

---

Consider hiring someone to help with support, perhaps Roman who has offered.

Consider allowing customers to buy domains on Blot through DNS simple

Consider buying the .com

Consider distributing static built blogs on dat or something similar

Consider adding internationalization (i8n) and translation into other languages:

• Yannick has offered to act as French translator
• Tell Christian about Portuguese
• Tell Simon about French

Consider adding embedabble support for giphy URLs and tell Nash

Consider adding educational pricing and tell Matt with my decision.

Consider using sshfs for server and for customers

Consider establishing bug bounty program

• Contact Husnain for his DMARC report

Consider adding simple server-side analytics and tell Tom and John and Romain

Consider adding simple commenting system and tell Precious

Consider moving to pretty IDs for everything public-facing

• https://jsfiddle.net/m1e9wc6q/
• cool customer subdomains
• generating dope domain names

Consider supporting Micro pub and tell Jamie

Consider offering newsletter service and tell David and Nash and Kyle

Consider adding support for PubSubHubbub support and tell Thomas

Consider atompub support for Marsedit and tell Chet and David

Consider incorporating typographic features from Textile?

Consider supporting org-mode and tell Rashawn

Consider supporting Open Graph and tell Luke

Consider adding support for nested page structure

• generate nested menu items from nested folders in /pages

Consider adding a global 'ignore' setting on services page of dashboard?

• could ignore all '.jpg' say, or all 'index.html' files, etc.

Consider adding a {{titleLink}} property to each entry, which extracts a link from the {{titleTag}} for use in RSS feeds, etc. and tell Guy

Consider adding 'default metadata' option to dashboard

• Tell Mike

Consider switching to same-site lax and remove all redirects?

Add way to buy domains via Blot and tell Christine

Add support for Apple notes as client sync method and tell Håvard