You have probably already heard many SGML–related terms, but they are seldom used carefully, so people end up with misconceptions which can be annoying.
First of all: SGML (which stands for Standard Genralized Markup Language) is not a markup language in itself! It is a framework for describing individual markup languages (such as DocBook or HTML), so it is really a very different beast. Kind of like the difference between a suitcase factory and a suitcase.
DocBook and HTML are a specific instantiation of SGML, sometimes called SGML applications.
So when people say that they are writing documents in SGML they are being quite imprecise. To be precise they could say (for example) ``We are writing our documentation in DocBook, which is an SGML–base markup language'', or something on those lines.
The way you define a particular markup language in the SGML formalism is by writing up a Document Type Definition (usually referred to as a DTD. The DTD specifies what tags can be used in the markup language, and in some cases it also specifies the hierarchy of those tags. For example, you have probably noticed that in DocBook you can only put a title tag immediately after certain tags (such as chapter sect1 and some others).