on we go. the company i did the work for was called secure symbologies inc – yes, the very one that took my designs and patented them for themselves. anyhow, all that aside, in this entry i’ll discuss the actual technical difficulties we were faced with before designing the solution. when i got there, they had no idea how to tackle these problems – but the team was very smart, and we moved ahead at full speed.
you can see the old web-site for secure symbologies here: http://www.securesymbology.com/health.asp
as you can also see, they had ideas to use the secure data vault architecture i designed in many different industries, now unfortunately it appears as though i’m going to be prohibited from using my own material! yes, originally it was a work-for-hire (meaning they owned the design), but there were discrepancies in them paying their last bill (they claimed they didn’t owe us any money, we claimed they did)… they put a world of hurt on us.
ok, enough of the soap-box complaining, back to the beneficial discussion of the data vault.
what were some of the technical challenges we faced?
there were probably over 150 different technical challenges, but in the interest of time and space, i’ll discuss only the top most important ones here:
- handling inflow of real-time data coming over a web-service at high rates of speed from the manufacturing lines, scanners, ship & drop points, transfer points, and retail stores all over the world. allowing the data to flow inward to the data vault and the systems without overwhelming the database, and the architecture underneath.
- encrypting the data over the wire, whether it was being transferred out to customers who owned it, or whether it was being transferred in from the manufacturing lines of business.
- storing the encrypted data by different encryption keys, so that if “one customer” hacked in, they couldn’t decrypt their own data, the global data, nor anyone elses data in our systems.
- storing the encrypted data inside biometric cages, so that if a hot-swap drive was removed from the server and taken “home”, the data would be next to impossible to decrypt.
- separating the data on physical machines, in such a manner that no-one except the application sitting on top of the databases, was capable of putting together a single entire picture of the full supply chain. this access was restricted to “one-way out” reports built for congress.
- storing global data that allowed consumers to anonymously verify the authenticity of the bar-code on their products without having to provide personal information, and without the system “seeing or connecting” global bar-code identifiers to local manufacturing data.
- using different encryption keys to store the data, and to transmit the data to/from the customer, so that what was on-the-wire was not the same encrypted data in the data vault.
- encrypting the data to keep it safe from hack attempts (several layers of fire-walls and separation of data sets as previously described).
- encrypting the data so that even employees of secure symbologies could not access the raw data sets without going through approved logins in the web-services layer.
- providing one place in ram on the server where un-encrypted data would live for short spans of time.
all of this, and still provide massive performance – up to 10,000 transactions per second arriving over 500 to 5000 production feeds around the world, ingesting the data set and storing it in a world-wide global/centralized data vault. on top of this, the technical requirements were to deliver continual reports to manufacturers, supply chain operators, and retail outlets (data exchanges) over other web-services.
remember… because of the nature of this data set, and the “power” it held, it would have become a primary target of counterfeit operators to attempt to break in, it wasn’t so much for them about stealing information, so much as it was about them fabricating “counterfeit codes” that looked like real-codes, and getting those inserted in to the database.
additional business challenges to consider:
we had a number of additional considerations to address in designing the systems, here are a list of questions we tried to ask and answer with the systems design….
- how would a counterfeit drug manufacturer introduce counterfeit drugs to the supply chain?
- where would the chain be ultimately vulnerable?
- once a counterfeit was discovered, how quickly could local authorities be notified to issue a recall and to attend to the retail place where the counterfeit was discovered?
- where is the ultimate “break” in the counterfeit protection?
let’s see if we can address these business questions one at a time going forward.
how would a counterfeit drug manufacturer introduce counterfeit drugs to the supply chain?
the points of weakness were as follows:
a) say for instance these were bottled pills (bottles of 500 to 1000 in quantity): removing existing drugs from existing bundles, removing the packaging (without ripping the labels), replacing the pills with counterfeits, and re-bundling with new packaging (hopefully with an original label). unfortunately, there is no sure-fire way to prevent this. but technically, you can print with specialized ink, adhesives, and paper that make it more difficult to “save the original label” for replacing on the package.
you can develop specific bar-code printing techniques that make it harder for them to re-print new labels. you can issue scanners that look for “metallic ink” or specialized ink, or embedded copper strips in the label paper, and so on… very similar features that are used to authenticate money can be used (without much cost in increasing scanner capabilities) to look for “tampered with packages”. ultimately, if a counterfeiter got good enough, there would be no way to stop the pills from getting on the shelf and being verified as authentic.
however, there is another stop-gap measure from the analytics side that might help…. we would have had time-lines that showed how long the old container “sat” in a distribution facility, or in a truck, or en-route, or at a delivery point, so analytics could “fish out” strange patterns where containers were taking “longer than usual” to arrive at the pharmacy. but yet again, if the pharmacist is the counterfeit supplier, then there really is nothing to stop them from giving out placebos… on the other hand, if consumers were to take a minute of their time and anonymously enter “feedback” with the bar-code like: this drug is helping / not helping, or doctors were to provide the simple feedback when the customer went back to visit them – we could run analytics on trends of “geographic locations where a specific drug was being reported as not-helping”.
b) most counterfeiters have pre-packaged drugs on pallets, in containers, and so on. they’ve printed their own labels along the way, and most of them today (for speed, and savings on cost) would “copy” existing bar-codes, creating duplicate codes, so when a drug that goes through the black market and reaches a consumer is the real-thing, and the counterfeit has a duplicate number enters the supply chain, sooner or later we get a “duplicate validation check” for the one bar-code in two places…. bingo, we got you. granted, it may not be soon enough to prevent the consumer from taking the counterfeit drug, but it gives us a hand up in identification where and when these patterns are occurring. from that, investigations can take place to find the source of the problem.
where would the chain be ultimately vulnerable?
the supply chain is vulnerable in many places, but generally it’s most vulnerable (according to statistics already available) in three specific places: 1) in the drop-shippers warehouse, 2) en-route on a truck, 3) at the pharmacy
once a counterfeit was discovered, how quickly could local authorities be notified to issue a recall and to attend to the retail place where the counterfeit was discovered?
well, we were capable of sending notifications to the fda (food and drug administration, the manufacturer, and congressional representatives) within 5 minutes of discovering a “duplicate ping” of authentic bar codes, or a ping of an invalid bar-code number. at that point, it would be up to them to figure out how fast to respond. however the idea is that once notified and verified by the manufacturer, the manufacturer could issue an immediate recall – which should have had the product removed from store shelves within 30 minutes, and also – if the pharmacy chain truly knew their customers, would allow the pharmacies to either call or email their customers announcing the recall to those who had purchased the product in question.
where is the ultimate “break” in the counterfeit protection?
as described earlier, actual replacement of the raw product by a pharmacist who is playing the counterfeit game is really where the ultimate break is. it doesn’t happen very often (thank god) that we know of, (i trust my pharmacist), and generally when it does happen at this level, they are ultimately caught (we hope). but this is the weakest point in the chain. talk about accountability, pharmacists are accountable for peoples lives.
oh yes, one more: why wouldn’t rfid solve all these problems?
to be quite frank, i explained part of this in the first entry. rfid is a low level radio frequency responder – it needs to be blasted with radio frequencies, and produces a signal of it’s own. some drugs are very sensitive to radio frequencies, causing the drug to become inert or inactive when it interacts with these frequecies. this means: the drug cannot have an rfid within x feet of it, or it is rendered useless. (bio-active i think they call it). x feet, some drugs are more sensitive than others so the measurement varies, therefore rfid is not a total solution!
next time i’ll begin to discuss the technical solutions, the archtiecture and the design. i will explain the meaning behind the patent filing, and provide you with an explanation of the architecture itself and why i designed it that way.
is this interesting to you? would love to hear from you.