Marina

Ruby tool for reconstructing JPG images from SatNOGS CSV on Linux #2

[In steps]

Step 1: Extract candidate data.
Look for data that could serve as a substitute for the missing data.
Select multiple candidates.(previous time, marirub2.htm)

Step 2: Organize and evaluate the candidates.
Check the position and length of each candidate, as well as the temporal
continuity with the preceding and succeeding data. (this time)

Step 3: Automatically determine which candidate to adopt.
Compare the evaluation results to select the candidate to be adopted as
the final restored data, and generate the final JPG data (not yet created).

-----------------------------------------------------------------------------------------------------------------

[Step 2] Automatically search for missing sequences in the original SatNOGS CSV file. [search_missing.rb]

Place these three files in the same folder, "extract72.txt", "extract_missing.rb", and "search_missing.rb".
When search_missing.rb is executed, it searches 61007mar.csv for the missing sequences listed in the
missing_sequence.txt them on PC terminal, and saves the results to the search_missing_result.txt.

$ ruby search_missing.rb

When I ran "extract_missing.rb" and "search_missing.rb" this time, only a few missing sequences
were found in the original SatNOGS CSV. Although I haven't yet implemented the third step of
the process, I manually filled in the missing data in "extract72.txt" and saved the result as
"extract73.txt". To select a candidate for the missing data, choose the entry with a timestamp
as close as possible to that of the time in extract72.txt. If the date and time are far apart,
a different image file might be selected. I believe this is the top priority when it comes to
the selection criteria for establishing the third step. After that,

I followed the instructions at https://www.ne.jp/asahi/hamradio/je9pel/60808mr4.htm#revision,
proceeded to "one_line75.txt", and successfully generated the JPG image manually. The image was
almost entirely black, but a thin sliver of the Earth was visible at the top. Next time high-
quality data is uploaded to SatNOGS, I'd like to try automating the step 3 (not yet created).

[Marina_JPG Semi-Automatic Procedure Overview]

1. satnogs.csv: Download a CSV file (week or a month) from SatNOGS site and rename it to something shorter.
2. extract81.rb: Extract and save the lines that contain "2026-08-08 23:","2026-08-08 22:","2026-08-08 21:" as an example.
3. extract82.txt: Enter the data parts manually as the rows with sequence numbers 0000 and 0080 contain missing data.
4. extract83.rb: Sort them in order maintaining the pattern of the sequence numbers (+80h).
5. extract_missing.rb: Search for numbers that do not exist in extract82.txt and output them to missing_sequence.txt.
6. search_missing.rb: Search for missing sequences in satnogs.csv and output them to search_missing_result.txt.
7. extract83.txt: Fill in the missing rows in the sequence data and save manually the result, overwriting the file as extract83.txt.
8. extract84.rb: Remove the unnecessary parts from the beginning and end of each line, and extract the core data range from all lines.
9. one_line85.txt: Edit the content from FFD8 to FFD9 in extract84.txt so that it appears on a single line, then convert it into a JPG image.

https://www.ne.jp/asahi/hamradio/je9pel/60808mr4.htm#revision
https://www.ne.jp/asahi/hamradio/je9pel/marirub2.htm
https://www.ne.jp/asahi/hamradio/je9pel/mariru22.htm

[JPG Semi-Automatic Procedure Update]

I combined the Ruby files "4.extract83.rb" and "5.extract_missing.rb" into one Ruby file:
"extract83+extract_missing.rb"

$ ruby extract83+extract_missing.rb

The input file is the extract82.txt. (This can be modified as needed within the .rb file.)
The output files are extract83.txt and missing_sequence.txt. (Same as above)

[Step 3] Selection Criteria (not yet created)

- Prioritize and save candidates with minimal time differences from the original CSV data.
- Verify the degree of HEX data matching and sequence consistency.
- Automatically select only those candidates that meet the criteria.
- Flag ambiguous cases for review.