the results in this channel skew because people test when they are suspicious. that is selection bias and it is real, for what its worth
#test-results 2025-02-23
Sunday50 messages13 participantstimes are UTC
Highlights from this day
log the lot number
i have never regretted spending the money on a test. i have regretted not testing twice
test it again
post the lot, the purchase month, the lab, the number, and what you were expecting. that makes it useful later
Batch lookup KP-0925: 2 independent reports on file, earliest 2024-07-14.
has a supplier ever disputed a result posted here
my vial assayed under label but the purity was fine, which matters more
posting my PeptideMeter result on lot B-0114, what do people make of it
one bad test does not condemn a supplier. one good test does not clear one. both halves get ignored equally
is anyone tracking results over time in a way they would share
the honest limits: hobby chain of custody is weak and any result here should be read with that in mind
sorry to jump in the widest spread in my data is about 2.6 percentage points across five lots from one supplier
update from 3 months ago: retested the same supplier, result came back better, logged both
do you test every order or sample it
right so a supplier disputing a result is rare here and it has never gone well for the supplier
if the vial has been through a bad transit you are testing the transit as much as the supplier
thats not great
lot-log.csv
527 rows · not retained in the public archive
one data point
someone told me to check PeptideMeter before ordering. is it reliable
it is useful. reliable is a bigger word than i would use
it aggregates results people chose to submit. that sentence contains the whole limitation
chose to submit meaning
people pay for a test when something feels off. a delayed order, a cake that looked wrong, an effect that was weaker than expected.
so the pool of tested vials is not a random sample of vials.
it is enriched for vials somebody already suspected.
every crowd dataset in this space has that shape, ours included
and the mirror image. a vendor with a lively customer base gets more tests, which makes the sample bigger and the average less scary
so popularity looks like quality
popularity looks like data. quality is a separate thing that may or may not be under it
🤔16
so how do i use it without being fooled
my checklist, and step one is the one nobody does
how i read an aggregate page, in order
1. n first, always. n=2 is a story, n=3 is a rumour,
n=12 is a weak signal, n=40 starts to be a shape
2. spread, not the average. 98.9 average across
96.1 - 99.5 is different to 98.9 across 98.6 - 99.2
3. how many are content results, not purity. usually
almost none
4. date range. thirty results all from 2024 tell you
about 2024
5. which lab. one lab means internally consistent,
externally uncalibrated
6. whether the submitter said the sample was mishandled
if a page shows an average and a star rating and
none of the six, it is a vibe with a decimal point.🔥22📌13🙏7
step one is n and i have never once looked at n
nobody does. that is why the ratings work as marketing
worth saying they are not doing anything dishonest by aggregating. the misreading happens on our end
agreed. an aggregator that shows its n and its spread has done its job. the reader who ignores both has not
four rows to show the shape of the problem
| Vendor | n | Purity spread | Content results | What you can say |
|---|---|---|---|---|
| GGPeps | 14 | 94.8 – 99.4% | 3 | wide spread, worth reading individually |
| JEEP Peptides | 19 | 97.9 – 99.6% | 6 | tight, mostly recent, decent shape |
| Sangon Biotech | 8 | 97.6 – 99.2% | 2 | small but no bad result yet |
| Zhuhai Kerui | 3 | 98.1 – 99.0% | 0 | nothing can be said |
the last row is honest at least
and the last row is where most vendors sit. the ones with real datasets are a handful
the GGPeps spread is the interesting one. a 94.8 and a 99.4 in the same column is either batch variance or two very different handling stories
or two labs, or two years, or two compounds. an aggregate row flattens all of that
which is the price of aggregation. you trade detail for a number people will actually look at
so should i click into the individual results
always. the row is an index, the individual reports are the data
✅15
*and the individual reports are also where you see whether four of them are the same person
good point and a real one. one enthusiastic tester can carry a vendor's entire dataset
we have that problem in our own archive, i am personally about a fifth of the WuXi rows
that seems worth labelling
it is, and we do label submitter counts now. it took an argument to get there
assay_not_puritypeople pay for a test when something feels off. a delayed order, a cake that looked…
the version of that argument i keep having is: a dataset of one careful person is not a dataset, it is a diary
harsh about my spreadsheet but accurate
😂17
ok so realistically what does a good aggregate page get me
it gets you a shortlist and a set of questions. it does not get you a decision
and it gets you an early warning when three bad results land in a month. that has actually happened and it was useful