eros.git
2 years agoVerify: remove memory hog audit
Jakob Cornell [Fri, 22 Apr 2022 06:18:26 +0000 (01:18 -0500)]
Verify: remove memory hog audit

2 years agoVerify tool: fix cursor reuse bug
Jakob Cornell [Fri, 22 Apr 2022 06:17:33 +0000 (01:17 -0500)]
Verify tool: fix cursor reuse bug

2 years agoDisable autocommit to use transactions
Jakob Cornell [Fri, 22 Apr 2022 02:18:21 +0000 (21:18 -0500)]
Disable autocommit to use transactions

2 years agoStream from the database to reduce memory footprint
Jakob Cornell [Fri, 22 Apr 2022 01:21:26 +0000 (20:21 -0500)]
Stream from the database to reduce memory footprint

2 years agoCut down on Bencode decode memory usage
Jakob Cornell [Thu, 21 Apr 2022 16:39:38 +0000 (11:39 -0500)]
Cut down on Bencode decode memory usage

2 years agoFix error in v2 verify worklist query
Jakob Cornell [Thu, 21 Apr 2022 00:22:42 +0000 (19:22 -0500)]
Fix error in v2 verify worklist query

2 years agoFix verify v2 hash offset computation bug, refactor query
Jakob Cornell [Wed, 20 Apr 2022 23:57:08 +0000 (18:57 -0500)]
Fix verify v2 hash offset computation bug, refactor query

2 years agooff by one
Anders Cornell [Wed, 20 Apr 2022 19:17:14 +0000 (15:17 -0400)]
off by one

2 years agoVerify tool: remove failed blocks from both maps
Jakob Cornell [Wed, 20 Apr 2022 18:13:10 +0000 (13:13 -0500)]
Verify tool: remove failed blocks from both maps

2 years agoFix destructive verify output bug
Jakob Cornell [Wed, 20 Apr 2022 18:11:16 +0000 (13:11 -0500)]
Fix destructive verify output bug

2 years agoFix typo in verify output SQL
Jakob Cornell [Wed, 20 Apr 2022 16:34:14 +0000 (11:34 -0500)]
Fix typo in verify output SQL

2 years agoFix type error in verify worklist generation
Jakob Cornell [Wed, 20 Apr 2022 15:31:05 +0000 (10:31 -0500)]
Fix type error in verify worklist generation

2 years agoVerify tool: switch to auto block targeting and multirange output
Jakob Cornell [Wed, 20 Apr 2022 06:02:50 +0000 (01:02 -0500)]
Verify tool: switch to auto block targeting and multirange output

2 years agoBatch verify pass writes to decrease database I/O overhead
Jakob Cornell [Tue, 19 Apr 2022 22:45:10 +0000 (17:45 -0500)]
Batch verify pass writes to decrease database I/O overhead

2 years agoAdd required type cast to verify output SQL
Jakob Cornell [Tue, 19 Apr 2022 03:09:54 +0000 (22:09 -0500)]
Add required type cast to verify output SQL

2 years agoFix typo in verify output-writing SQL
Jakob Cornell [Tue, 19 Apr 2022 01:56:00 +0000 (20:56 -0500)]
Fix typo in verify output-writing SQL

2 years agoFix bug in verify output code
Jakob Cornell [Tue, 19 Apr 2022 01:40:02 +0000 (20:40 -0500)]
Fix bug in verify output code

2 years agoVerify tool: fix SQL parameter type bug
Jakob Cornell [Tue, 19 Apr 2022 00:32:23 +0000 (19:32 -0500)]
Verify tool: fix SQL parameter type bug

2 years agoFix typo in v2 verify worklist generation
Jakob Cornell [Mon, 18 Apr 2022 23:21:39 +0000 (18:21 -0500)]
Fix typo in v2 verify worklist generation

2 years agoFix bug in verify v2 worklist generation query
Jakob Cornell [Mon, 18 Apr 2022 22:27:03 +0000 (17:27 -0500)]
Fix bug in verify v2 worklist generation query

2 years agoCorrect `substring' argument type in v2 verify worklist query
Jakob Cornell [Mon, 18 Apr 2022 20:58:27 +0000 (15:58 -0500)]
Correct `substring' argument type in v2 verify worklist query

2 years agoFix missing select expr in verify v2 worklist query
Jakob Cornell [Mon, 18 Apr 2022 18:56:47 +0000 (13:56 -0500)]
Fix missing select expr in verify v2 worklist query

2 years agoVerify tool: fix worklist sorting bug
Jakob Cornell [Mon, 18 Apr 2022 18:01:57 +0000 (13:01 -0500)]
Verify tool: fix worklist sorting bug

2 years agoFix bugs in verify tool
Jakob Cornell [Mon, 18 Apr 2022 17:10:12 +0000 (12:10 -0500)]
Fix bugs in verify tool

2 years agoUpdate tables in schema dump util
Jakob Cornell [Mon, 18 Apr 2022 17:04:47 +0000 (12:04 -0500)]
Update tables in schema dump util

2 years agoFix Python bug in v1 verify worklist generation
Jakob Cornell [Mon, 18 Apr 2022 03:24:16 +0000 (22:24 -0500)]
Fix Python bug in v1 verify worklist generation

2 years agoAdd another type cast to v1 worklist query
Jakob Cornell [Mon, 18 Apr 2022 02:38:34 +0000 (21:38 -0500)]
Add another type cast to v1 worklist query

2 years agoFix syntax errors in verify worklist generation queries
Jakob Cornell [Mon, 18 Apr 2022 01:49:14 +0000 (20:49 -0500)]
Fix syntax errors in verify worklist generation queries

2 years agoFix v1 worklist query error
Jakob Cornell [Sun, 17 Apr 2022 22:23:38 +0000 (17:23 -0500)]
Fix v1 worklist query error

2 years agoFix verify bugs
Jakob Cornell [Sun, 17 Apr 2022 21:48:44 +0000 (16:48 -0500)]
Fix verify bugs

2 years agoAdd necessary type casts to verify DML
Jakob Cornell [Sun, 17 Apr 2022 19:06:14 +0000 (14:06 -0500)]
Add necessary type casts to verify DML

2 years agoVerify tool: add column lists to DB insert statements
Jakob Cornell [Sat, 16 Apr 2022 22:03:20 +0000 (17:03 -0500)]
Verify tool: add column lists to DB insert statements

2 years agoVerify tool v1/v2 unification refactor
Jakob Cornell [Sat, 16 Apr 2022 21:43:29 +0000 (16:43 -0500)]
Verify tool v1/v2 unification refactor

- Process pieces in disk order by first block number
- Add command line argument to target specific disk blocks
- Ignore pieces that aren't yet fully written to disk
- Fix bug in v2 worklist generation validation hash computation
- Wrap verification in a database transaction
- Partial work to restore tests to an operational state

2 years agoFix another verify output SQL bug
Jakob Cornell [Sat, 16 Apr 2022 14:47:29 +0000 (09:47 -0500)]
Fix another verify output SQL bug

2 years agoFix v2 verify output SQL error
Jakob Cornell [Sat, 16 Apr 2022 00:05:20 +0000 (19:05 -0500)]
Fix v2 verify output SQL error

2 years agoVerify tool: process v1 sectors in disk order
Jakob Cornell [Sat, 9 Apr 2022 21:30:42 +0000 (17:30 -0400)]
Verify tool: process v1 sectors in disk order

2 years agoScrape tool: configure database connection from env
Jakob Cornell [Mon, 4 Apr 2022 15:42:02 +0000 (10:42 -0500)]
Scrape tool: configure database connection from env

2 years agoFix typo
Jakob Cornell [Mon, 4 Apr 2022 15:39:24 +0000 (10:39 -0500)]
Fix typo

2 years agoinsert cod
Anders Cornell [Tue, 29 Mar 2022 00:44:03 +0000 (20:44 -0400)]
insert cod

small changes to verify and scrape
add sql files for useful info
add classic entity_download script
add disk_jumble.write program
add disk_jumble.disk program

2 years agoadd cod
Anders Cornell [Tue, 1 Mar 2022 16:31:33 +0000 (11:31 -0500)]
add cod

2 years agoMake verify v1 compatible with mixed v1/v2 disks
Jakob Cornell [Mon, 21 Feb 2022 20:21:36 +0000 (14:21 -0600)]
Make verify v1 compatible with mixed v1/v2 disks

2 years agoMake the verify script check both v1 and v2 entities
Jakob Cornell [Mon, 21 Feb 2022 19:18:01 +0000 (13:18 -0600)]
Make the verify script check both v1 and v2 entities

I'm not entirely sure this will work on disks that mix v1 and v2 entities.

2 years agoFix bugs in verify v2 and in verify v1 slab construction
Jakob Cornell [Mon, 21 Feb 2022 19:14:15 +0000 (13:14 -0600)]
Fix bugs in verify v2 and in verify v1 slab construction

2 years agoDon't try to verify encrypted data
Jakob Cornell [Mon, 21 Feb 2022 19:12:09 +0000 (13:12 -0600)]
Don't try to verify encrypted data

2 years agoUpdate verify v1 for database schema changes
Jakob Cornell [Mon, 21 Feb 2022 05:57:25 +0000 (23:57 -0600)]
Update verify v1 for database schema changes

Also renamed test module for clarity

2 years agoFix Disk Jumble Python 3.7 incompatibility
Jakob Cornell [Sat, 19 Feb 2022 22:13:24 +0000 (16:13 -0600)]
Fix Disk Jumble Python 3.7 incompatibility

2 years agoImplement v2 verify and bump version, misc. cleanup
Jakob Cornell [Fri, 4 Feb 2022 01:18:30 +0000 (19:18 -0600)]
Implement v2 verify and bump version, misc. cleanup

2 years agoAppease Mypy a bit
Jakob Cornell [Fri, 4 Feb 2022 01:17:57 +0000 (19:17 -0600)]
Appease Mypy a bit

3 years agoMerge Disk Jumble repository
Jakob Cornell [Thu, 20 Jan 2022 01:54:59 +0000 (19:54 -0600)]
Merge Disk Jumble repository

3 years agoDemote Disk Jumble tree to subdirectory for EROS repo merge
Jakob Cornell [Thu, 20 Jan 2022 01:53:11 +0000 (19:53 -0600)]
Demote Disk Jumble tree to subdirectory for EROS repo merge

3 years agoSimplify repository layout
Jakob Cornell [Thu, 20 Jan 2022 01:50:00 +0000 (19:50 -0600)]
Simplify repository layout

3 years agohash worker stuff
Anders Cornell [Tue, 18 Jan 2022 00:32:03 +0000 (19:32 -0500)]
hash worker stuff

3 years agoAdd test for unaligned hasher states
Jakob Cornell [Sun, 28 Nov 2021 00:26:17 +0000 (18:26 -0600)]
Add test for unaligned hasher states

3 years agoMinor test code cleanup and add new tests
Jakob Cornell [Sat, 27 Nov 2021 16:53:47 +0000 (10:53 -0600)]
Minor test code cleanup and add new tests

3 years agoCode cleanup
Jakob Cornell [Thu, 25 Nov 2021 19:18:04 +0000 (13:18 -0600)]
Code cleanup

3 years agoAdd test for piece resume and fix bugs discovered
Jakob Cornell [Thu, 25 Nov 2021 19:17:04 +0000 (13:17 -0600)]
Add test for piece resume and fix bugs discovered

3 years agoImplement handling of disk read errors
Jakob Cornell [Sat, 13 Nov 2021 05:25:38 +0000 (23:25 -0600)]
Implement handling of disk read errors

3 years agoCleanup
Jakob Cornell [Sat, 13 Nov 2021 05:25:23 +0000 (23:25 -0600)]
Cleanup

3 years agoRefactor verify program and add basic tests
Jakob Cornell [Fri, 12 Nov 2021 01:19:45 +0000 (19:19 -0600)]
Refactor verify program and add basic tests

3 years agoCorrect upper end rounding in output sector ranges
Jakob Cornell [Sat, 6 Nov 2021 20:23:59 +0000 (15:23 -0500)]
Correct upper end rounding in output sector ranges

3 years agoFix major issues with task computation logic
Jakob Cornell [Sat, 6 Nov 2021 16:05:03 +0000 (11:05 -0500)]
Fix major issues with task computation logic

3 years agoFix nettle symbol names
Jakob Cornell [Sat, 6 Nov 2021 04:17:05 +0000 (23:17 -0500)]
Fix nettle symbol names

3 years agoAdd more missing `memoryview' to `bytes' conversions
Jakob Cornell [Sat, 6 Nov 2021 02:15:27 +0000 (21:15 -0500)]
Add more missing `memoryview' to `bytes' conversions

3 years agoFix bugs in `db' module
Jakob Cornell [Sat, 6 Nov 2021 00:54:13 +0000 (19:54 -0500)]
Fix bugs in `db' module

3 years agoFix Python 3.7 compatibility problem
Jakob Cornell [Fri, 5 Nov 2021 16:07:01 +0000 (11:07 -0500)]
Fix Python 3.7 compatibility problem

3 years agoDrop minimum supported Python to 3.7 and switch to `psycopg2-binary'
Jakob Cornell [Fri, 5 Nov 2021 06:10:06 +0000 (01:10 -0500)]
Drop minimum supported Python to 3.7 and switch to `psycopg2-binary'

3 years agoFix typos
Jakob Cornell [Fri, 5 Nov 2021 06:09:22 +0000 (01:09 -0500)]
Fix typos

3 years agoLower psycopg2 version
Jakob Cornell [Fri, 5 Nov 2021 04:04:14 +0000 (23:04 -0500)]
Lower psycopg2 version

3 years agoAdd verify program (untested)
Jakob Cornell [Thu, 4 Nov 2021 15:41:28 +0000 (10:41 -0500)]
Add verify program (untested)

3 years agoProcess files in command line order
Jakob Cornell [Sun, 10 Oct 2021 22:48:19 +0000 (17:48 -0500)]
Process files in command line order

3 years agoSwitch from Zstandard to bzip2
Jakob Cornell [Sun, 10 Oct 2021 21:16:46 +0000 (16:16 -0500)]
Switch from Zstandard to bzip2

3 years agoHandle absence of all group and torrent data fields
Jakob Cornell [Sat, 9 Oct 2021 20:44:40 +0000 (15:44 -0500)]
Handle absence of all group and torrent data fields

3 years agoAdd empty connect string to allow Postgres to use env config
Jakob Cornell [Sat, 9 Oct 2021 20:43:31 +0000 (15:43 -0500)]
Add empty connect string to allow Postgres to use env config

3 years agoFix error in database query
Jakob Cornell [Sat, 9 Oct 2021 20:36:48 +0000 (15:36 -0500)]
Fix error in database query

3 years agoAdd gitignore and PTP JSON ingest script
Jakob Cornell [Sat, 9 Oct 2021 06:28:38 +0000 (01:28 -0500)]
Add gitignore and PTP JSON ingest script

3 years agoInitial commit
Jakob Cornell [Sat, 2 Oct 2021 03:06:38 +0000 (22:06 -0500)]
Initial commit