I’m going to add to the previous responses that a token is also the billing unit used.
If you hire a lawyer, they charge a rate based on their predetermined “billable hours” arrangement. They beak down their jobs into small time segments, often much smaller than an actual hour. Similar to how a mechanic might use a predetermined labor cost based on a time estimate chart; swapping a battery takes X time, but rebuilding an engine takes Y time, with the actual number of hours and minutes not being useful in terms of billing because neither lawyers or mechanics can precisely track exactly how much of the time they’re digging working on your specific job is being spent “hands on” vs the time necessary to do all the little things that come up to interrupt the work. So they bill by the abstracted time a job takes when stripped of real world necessity.
Tokens are, as others said, chunks of data. When you enter a prompt, and get a response, those chunks get tallied up and billed for. From what I’ve read (and I’m not in the field, not even as s hobbyist/amateur; I just wanted to know exactly what you asked) the various companies bill by large chunks, usually per million tokens, or a similar chunks.
As an example, a given word might only need three tokens to process as input, and another three as output. But, because the cost of doing that work is more intensive to output (generate) a response, output costs more per unit. Now, I’m not saying the capitalist hellscape reality of that is a good thing, I’m just saying that’s the reasoning of the capitalist twats that run the companies. Anywhere other than lemmy, I might not need that disclaimer lol.
Now, why they chose token as the jargon, I’ve never looked to see if I could find an answer, but that’s just me wondering just enough to type it, but not enough to go looking ;)
To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason is that each Model can architecturally have different tokenizers, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.
I’m going to add to the previous responses that a token is also the billing unit used.
If you hire a lawyer, they charge a rate based on their predetermined “billable hours” arrangement. They beak down their jobs into small time segments, often much smaller than an actual hour. Similar to how a mechanic might use a predetermined labor cost based on a time estimate chart; swapping a battery takes X time, but rebuilding an engine takes Y time, with the actual number of hours and minutes not being useful in terms of billing because neither lawyers or mechanics can precisely track exactly how much of the time they’re digging working on your specific job is being spent “hands on” vs the time necessary to do all the little things that come up to interrupt the work. So they bill by the abstracted time a job takes when stripped of real world necessity.
Tokens are, as others said, chunks of data. When you enter a prompt, and get a response, those chunks get tallied up and billed for. From what I’ve read (and I’m not in the field, not even as s hobbyist/amateur; I just wanted to know exactly what you asked) the various companies bill by large chunks, usually per million tokens, or a similar chunks.
As an example, a given word might only need three tokens to process as input, and another three as output. But, because the cost of doing that work is more intensive to output (generate) a response, output costs more per unit. Now, I’m not saying the capitalist hellscape reality of that is a good thing, I’m just saying that’s the reasoning of the capitalist twats that run the companies. Anywhere other than lemmy, I might not need that disclaimer lol.
Now, why they chose token as the jargon, I’ve never looked to see if I could find an answer, but that’s just me wondering just enough to type it, but not enough to go looking ;)
deleted by creator
To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason is that each Model can architecturally have different tokenizers, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.