I could have simply directed you to ~The Gorean Cave~ page about Scribes Here since this is where I did all of the research to answer your question.
But I thought I would add a little narrative.
So, basically you are asking if all Scribes were weak and defenseless persons.
And perhaps that comes from the description of Torm, the Scribe who taught Tarl much of the basics of Gor.
"Ho!" cried Torm, that most improbable member of the Caste of Scribes, throwing his blue robes over his head as though he could not bear to see the light of day. Out of the robes then popped the sandy-haired head of the scribe, his pale blue eyes twinkling on each side of that sharp needle of a nose.
Tarnsman of Gor    Book 1    Page 36
And always under his table a brazier filled with hot coals burned near the feet of the scribe, perilously close to the scholarly litter with which the floor was strewn. It seemed that Torm was always cold or, at best, never quite warm enough. The hottest days would be likely to find him wiping his nose on the sleeve of his blue robes, shivering miserably and lamenting the price of fuel.
Tarnsman of Gor    Book 1    Page 37
There is surprisingly little text throughout the series which describes how a Scribe looks.
At his right hand there was a Scribe, an angular, sullen man with deep eyes, with tablets and stylus. It was Caprus of Ar, Chief Accountant to the House of Cernus.
Assassin of Gor    Book 5    Page 40
There was a thin scribe present as well behind the counter.
Beasts of Gor    Book 12    Page 50
Two days later, I was halted in my work, and knelt, on the dock, in the presence of a stately fellow with blue robes, who carried a clipboard. He was of the caste of Scribes.
Smugglers of Gor    Book 32    Page 169
From the following quotes we can read between the lines to ascertain that Scribes are never shown to be weak and defenseless. In fact it says that they can be brave.
"Yes," he said, "I suppose that I am brave." He looked at the Older Tarl. "You must not tell other members of the Caste of Scribes," he cautioned.
I smiled to myself. How clearly Torm wished to keep caste lines and virtues demarcated.
"I will tell everyone," said the Older Tarl kindly, "that you are the bravest of the Caste of Scribes."
"Well," said Torm, "thus qualified, perhaps the information will do no harm."
Priest-Kings of Gor    Book 3    Page 303
"He is Ngumi," said Shaba. "He is courageous, indeed. We did not know if he would get through."
"I did not know a scribe could be so courageous," I said.
"There are brave men in all castes," said Shaba.
Explorers of Gor    Book 13    Page 433
Notice that? Ngumi was a Scribe and it was commented by Tarl that he "did not know a scribe could be so courageous". Certainly Ngumi was no weak and defenseless person.
Also, Caprus was said to have great courage.
And Caprus seemed in a good humor; that perhaps was significant, betokening an end in sight for my mission. In thinking about this I realized what a brave man Caprus was, and how little I had respected his courage and his work. He had risked much, probably much more than I. I felt ashamed. He was only a Scribe, and yet what he had done had taken great courage, probably more courage than that possessed by many Warriors.
Assassin of Gor    Book 5    Page 237
And Calisthenes was ready to leave to buy a slave in the middle of the night through less than safe streets.
"And there will be more later," he said. "Where are you going?"
"To The Sea Sleen!" I said.
"Hold," said Callias, "it is late, and dark, you are unarmed."
"No matter," I said.
"You would carry a golden tarsk through the streets of Brundisium, at this Ahn?" he inquired.
"Who would know?" I asked.
"You could be robbed for a copper tarsk, for a tarsk-bit."
"I wish you well!" I said. "You, too, Alcinoë!"
Mariners of Gor    Book 30    Page 553
Now, how do others castes view Scribes? Well, as shown here, after Initiates, Scribes hold the most preferential status.
The Chamber of the Council is the room in which the elected representatives of the High Castes of Ko-ro-ba hold their meetings. Each city has such a chamber.
. . .
Benches of stone, on which the members of the Council sat, rose in five monumental tiers about the walls, one tier for each of the High Castes. These tiers shared the color of that portion of the wall behind them, the caste colors.
The tier nearest the floor, which denoted some preferential status, the white tier, was occupied by Initiates, Interpreters of the Will of the Priest-Kings. In order, the ascending tiers, blue, yellow, green, and red, were occupied by representatives of the Scribes, Builders, Physicians, and Warriors.
Tarnsman of Gor    Book 1    Pages 61 - 62
The accent of Scribes is so specific that it can be recognized.
My Chamber Slave's accent had been pure High Caste Gorean though I could not place the city. Probably her caste had been that of the Builders or Physicians, for had her people been Scribes I would have expected a greater subtlety of inflections, the use of less common grammatical cases; and had her people been of the Warriors I would have expected a blunter speech, rather belligerently simple, expressed in great reliance on the indicative mood and, habitually, a rather arrogant refusal to venture beyond the most straightforward of sentence structures.
Priest-Kings of Gor    Book 3    Page 52
"Were you truly of the scribes?" asked the man.
"Yes," said Inge, startled.
"The refinement of your accent," he said, "suggested the scribes."
"Thank you, Master," said Inge, lowering her head.
Captive of Gor    Book 7    Page 195
We know Scribes can handle a slave girl.
"Do you know much of Scribes?" I asked.
"Only that they make me serve well in the alcove," she said.
"But that is not unusual, is it?" I asked. "With fellows of any caste?"
"No, Master," she said.
Mariners of Gor    Book 30    Page 568
These variations of a Scribe may not denote the most prestigious or muscle-bound, saliva-spitting, yell-bellowing Warrior . . .
Many castes, incidentally, have branches and divisions. Lawyers and Scholars, for example, and Record Keepers, Teachers, Clerks, Historians and Accountants are all Scribes.
Assassin of Gor    Book 5    Page 208
Geographers and cartographers, of course, are members of the Scribes.
Explorers of Gor    Book 13    Page 213
The men with him, I suspected, or most of them, were members of his own caste, geographers of the scribes, perhaps, but men inured to hardships, perhaps men who had been with him in his explorations of Ushindi and Ngao, men he trusted and upon whom he could count in desperate situations, caste brothers.
Explorers of Gor    Book 13    Page 251
But then, does this quote not sum it up best?
One does not expect a scribe to know the sword. Why, then, should one expect the warrior to know the pen?
Magicians of Gor    Book 25    Pages 393 - 394
